{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trade Smarter w/ Reinforcement Learning\n",
    "## A deep dive into TensorTrade - the Python framework for trading and investing using deep reinforcement learning\n",
    "\n",
    "![Banner](img/banner.jpeg)\n",
    "\n",
    "Winning high stakes poker tournaments, beating world-class StarCraft players, and autonomously driving Tesla's futuristic sports cars. What do they all have in common? Each of these extremely complex tasks were long thought to be impossible by machines, until only recently being made possible through the massive advancements in deep reinforcement learning. \n",
    "\n",
    "Reinforcement learning is beginning to take over the world.\n",
    "\n",
    "![Source: https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii](img/alphastar.gif)\n",
    "\n",
    "A little over two months ago, I decided I wanted to take part in the revolution, so I set out on a journey to create a profitable Bitcoin trading strategy using state-of-the-art deep reinforcement learning algorithms. While I made quite a bit of progress on that front, I realized that the tooling for this sort of project can be quite daunting to wrap your head around, and as such, it is very easy to get lost in the details.\n",
    "\n",
    "In between optimizing my previous project for distributed high-performance computing (HPC) systems; getting lost in endless pipelines of data and feature optimizations; and running my head in circles around efficient model set-up, tuning, training, and evaluation; I realized that there had to be a better way of doing things. After countless hours of researching existing projects, spending endless nights watching PyData conference talks, and having many back-and-forth conversations with the hundreds of members of the  RL trading Discord community, I realized there weren't any existing solutions that were all that good.\n",
    "\n",
    "There were many bits and pieces of great reinforcement learning trading systems spread across the inter-webs, but nothing solid and complete. For this reason, I've decided to create an open source Python framework for getting any trading strategy from idea to production, efficiently, using deep reinforcement learning. \n",
    "\n",
    "Enter TensorTrade. The idea was to create a highly modular framework for building efficient reinforcement learning trading strategies in a composable, maintainable way. Sounds like a mouthful of buzz-words if you ask me, so let's get into the meat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "![Logo](img/logo.jpeg)\n",
    "\n",
    "TensorTrade is an open source Python framework for training, evaluating, and deploying robust trading strategies using deep reinforcement learning. The framework focuses on being highly composable and extensible, to allow the system to scale from simple trading strategies on a single CPU, to complex investment strategies run on a distribution of HPC machines.\n",
    "\n",
    "Under the hood, the framework uses many of the APIs from existing machine learning libraries to maintain high quality data pipelines and learning models. One of the main goals of TensorTrade is to enable fast experimentation with algorithmic trading strategies, by leveraging the existing tools and pipelines provided by `numpy`, `pandas`, `gym`, `keras`, and `tensorflow`.\n",
    "\n",
    "The aim is to simplify the process of testing and deploying robust trading agents using deep reinforcement learning, to allow you and I to focus on creating profitable strategies.\n",
    "\n",
    "## RL Primer\n",
    "\n",
    "In case your reinforcement learning chops are a bit rusty, let's quickly go over the basic concepts.\n",
    "\n",
    "Every reinforcement learning problem starts out with an environment and one or more agents that can interact with the environment.\n",
    "\n",
    "![Primer](img/primer.jpg)\n",
    "\n",
    "_This technique is based off Markov Decision Processes (MDP) dating back to the 1950s._\n",
    "\n",
    "The agent will first observe the environment, then build a model of the current state and the expected value of actions within that environment. Based on that model, the agent will then take the action it has deemed as having the highest expected value.\n",
    "\n",
    "Based on the effects of the chosen action within the environment, the agent will be rewarded by an size corresponding to the actual value of that action. The reinforcement learning agent can then, through the process of trial and error (i.e. learning through reinforcement), improve its underlying model and learn to take more rewarding actions over time.\n",
    "\n",
    "If you still need a bit of refreshment on the subject, there is a link to an article titled _Introduction to Deep Reinforcement Learning_ in the references for this article, which goes much more in-depth into the details. Let's move on.\n",
    "\n",
    "## Getting Started\n",
    "\n",
    "The following tutorial should provide enough examples to get you started with creating simple trading strategies using TensorTrade, although you will quickly see the framework is capable of handling much more complex configurations.\n",
    "\n",
    "## Installation\n",
    "\n",
    "TensorTrade requires Python 3.6 or later, so make sure you're using a valid version before pip installing the framework. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import warnings\n",
    "import numpy\n",
    "\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "\n",
    "warnings.warn = warn\n",
    "warnings.simplefilter(action='ignore', category=FutureWarning)\n",
    "numpy.seterr(divide = 'ignore') \n",
    "\n",
    "sys.path.append(os.path.dirname(os.path.abspath('')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To follow this entire tutorial, you will need to install some extra dependencies, such as `tensorflow`, `stable-baselines`, `tensorforce`, `ccxt`, and `stochastic`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Obtaining file:///Users/adam/Desktop/Hedger/tensortrade\n",
      "Requirement already satisfied, skipping upgrade: numpy==1.16.4 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (1.16.4)\n",
      "Requirement already satisfied, skipping upgrade: pandas==0.25.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (0.25.0)\n",
      "Requirement already satisfied, skipping upgrade: gym==0.14.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (0.14.0)\n",
      "Requirement already satisfied, skipping upgrade: stable-baselines==2.8.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (2.8.0)\n",
      "Requirement already satisfied, skipping upgrade: mpi4py in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (3.0.2)\n",
      "Requirement already satisfied, skipping upgrade: ccxt==1.18.1220 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (1.18.1220)\n",
      "Requirement already satisfied, skipping upgrade: stochastic==0.4.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (0.4.0)\n",
      "Requirement already satisfied, skipping upgrade: tensorforce==0.5.2 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (0.5.2)\n",
      "Requirement already satisfied, skipping upgrade: tensorflow==1.13.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensortrade==0.0.1a18) (1.13.1)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pandas==0.25.0->tensortrade==0.0.1a18) (2019.3)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pandas==0.25.0->tensortrade==0.0.1a18) (2.8.0)\n",
      "Requirement already satisfied, skipping upgrade: scipy in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from gym==0.14.0->tensortrade==0.0.1a18) (1.3.1)\n",
      "Requirement already satisfied, skipping upgrade: pyglet<=1.3.2,>=1.2.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from gym==0.14.0->tensortrade==0.0.1a18) (1.3.2)\n",
      "Requirement already satisfied, skipping upgrade: six in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from gym==0.14.0->tensortrade==0.0.1a18) (1.12.0)\n",
      "Requirement already satisfied, skipping upgrade: cloudpickle~=1.2.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from gym==0.14.0->tensortrade==0.0.1a18) (1.2.2)\n",
      "Requirement already satisfied, skipping upgrade: joblib in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from stable-baselines==2.8.0->tensortrade==0.0.1a18) (0.14.0)\n",
      "Requirement already satisfied, skipping upgrade: matplotlib in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from stable-baselines==2.8.0->tensortrade==0.0.1a18) (3.1.1)\n",
      "Requirement already satisfied, skipping upgrade: opencv-python in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from stable-baselines==2.8.0->tensortrade==0.0.1a18) (4.1.1.26)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2018.1.18 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (2019.9.11)\n",
      "Requirement already satisfied, skipping upgrade: aiodns==1.1.1; python_version >= \"3.5.2\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (1.1.1)\n",
      "Requirement already satisfied, skipping upgrade: yarl==1.1.0; python_version >= \"3.5.2\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: setuptools>=38.5.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (40.6.2)\n",
      "Requirement already satisfied, skipping upgrade: requests>=2.18.4 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (2.22.0)\n",
      "Requirement already satisfied, skipping upgrade: aiohttp>=3.0.1; python_version >= \"3.5.2\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (3.6.2)\n",
      "Requirement already satisfied, skipping upgrade: cryptography>=2.6.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from ccxt==1.18.1220->tensortrade==0.0.1a18) (2.8)\n",
      "Requirement already satisfied, skipping upgrade: tqdm in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorforce==0.5.2->tensortrade==0.0.1a18) (4.36.1)\n",
      "Requirement already satisfied, skipping upgrade: pytest in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorforce==0.5.2->tensortrade==0.0.1a18) (5.2.2)\n",
      "Requirement already satisfied, skipping upgrade: gast>=0.2.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (0.3.2)\n",
      "Requirement already satisfied, skipping upgrade: astor>=0.6.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (0.8.0)\n",
      "Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: wheel>=0.26 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (0.33.6)\n",
      "Requirement already satisfied, skipping upgrade: tensorflow-estimator<1.14.0rc0,>=1.13.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.13.0)\n",
      "Requirement already satisfied, skipping upgrade: tensorboard<1.14.0,>=1.13.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.13.1)\n",
      "Requirement already satisfied, skipping upgrade: keras-applications>=1.0.6 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.0.8)\n",
      "Requirement already satisfied, skipping upgrade: absl-py>=0.1.6 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (0.8.1)\n",
      "Requirement already satisfied, skipping upgrade: grpcio>=1.8.6 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: protobuf>=3.6.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow==1.13.1->tensortrade==0.0.1a18) (3.10.0)\n",
      "Requirement already satisfied, skipping upgrade: future in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pyglet<=1.3.2,>=1.2.0->gym==0.14.0->tensortrade==0.0.1a18) (0.18.1)\n",
      "Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from matplotlib->stable-baselines==2.8.0->tensortrade==0.0.1a18) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from matplotlib->stable-baselines==2.8.0->tensortrade==0.0.1a18) (2.4.2)\n",
      "Requirement already satisfied, skipping upgrade: cycler>=0.10 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from matplotlib->stable-baselines==2.8.0->tensortrade==0.0.1a18) (0.10.0)\n",
      "Requirement already satisfied, skipping upgrade: pycares>=1.0.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from aiodns==1.1.1; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (3.0.0)\n",
      "Requirement already satisfied, skipping upgrade: idna>=2.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from yarl==1.1.0; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (2.8)\n",
      "Requirement already satisfied, skipping upgrade: multidict>=4.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from yarl==1.1.0; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (4.5.2)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from requests>=2.18.4->ccxt==1.18.1220->tensortrade==0.0.1a18) (1.25.6)\n",
      "Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from requests>=2.18.4->ccxt==1.18.1220->tensortrade==0.0.1a18) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.5; python_version < \"3.7\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from aiohttp>=3.0.1; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (3.7.4)\n",
      "Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from aiohttp>=3.0.1; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (3.0.1)\n",
      "Requirement already satisfied, skipping upgrade: idna-ssl>=1.0; python_version < \"3.7\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from aiohttp>=3.0.1; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (1.1.0)\n",
      "Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from aiohttp>=3.0.1; python_version >= \"3.5.2\"->ccxt==1.18.1220->tensortrade==0.0.1a18) (19.3.0)\n",
      "Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from cryptography>=2.6.1->ccxt==1.18.1220->tensortrade==0.0.1a18) (1.13.1)\n",
      "Requirement already satisfied, skipping upgrade: py>=1.5.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (1.8.0)\n",
      "Requirement already satisfied, skipping upgrade: wcwidth in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (0.1.7)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata>=0.12; python_version < \"3.8\" in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (0.23)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied, skipping upgrade: pluggy<1.0,>=0.12 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (0.13.0)\n",
      "Requirement already satisfied, skipping upgrade: atomicwrites>=1.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (1.3.0)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (19.2)\n",
      "Requirement already satisfied, skipping upgrade: more-itertools>=4.0.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (7.2.0)\n",
      "Requirement already satisfied, skipping upgrade: mock>=2.0.0 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow==1.13.1->tensortrade==0.0.1a18) (3.0.5)\n",
      "Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1->tensortrade==0.0.1a18) (3.1.1)\n",
      "Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1->tensortrade==0.0.1a18) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: h5py in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from keras-applications>=1.0.6->tensorflow==1.13.1->tensortrade==0.0.1a18) (2.10.0)\n",
      "Requirement already satisfied, skipping upgrade: pycparser in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.6.1->ccxt==1.18.1220->tensortrade==0.0.1a18) (2.19)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /Users/adam/.pyenv/versions/3.6.9/lib/python3.6/site-packages (from importlib-metadata>=0.12; python_version < \"3.8\"->pytest->tensorforce==0.5.2->tensortrade==0.0.1a18) (0.6.0)\n",
      "Installing collected packages: tensortrade\n",
      "  Found existing installation: TensorTrade 0.0.1a18\n",
      "    Uninstalling TensorTrade-0.0.1a18:\n",
      "      Successfully uninstalled TensorTrade-0.0.1a18\n",
      "  Running setup.py develop for tensortrade\n",
      "Successfully installed tensortrade\n",
      "\u001b[33mYou are using pip version 18.1, however version 19.3.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -e ..[tf,tensorforce,baselines,ccxt,fbm] -U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all the installation necessary! Let's get into the code.\n",
    "\n",
    "# TensorTrade Components\n",
    "\n",
    "TensorTrade is built around modular components that together make up a trading strategy. Trading strategies combine reinforcement learning agents with composable trading logic in the form of a `gym` environment. A trading environment is made up of a set of modular components that can be mixed and matched to create highly diverse trading and investment strategies. I will explain this in further detail later, but for now it is enough to know the basics.\n",
    "\n",
    "![Components](img/components.jpeg)\n",
    "\n",
    "Just like electrical components, the purpose of TensorTrade components is to be able to mix and match them as necessary.\n",
    "\n",
    "_The code snippets in this section should serve as guidelines for creating new strategies and components. There will likely be missing implementation details that will become more clear in a later section, as more components are defined._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trading Environment\n",
    "\n",
    "A trading environment is a reinforcement learning environment that follows OpenAI's `gym.Env` specification. This allows us to leverage many of the existing reinforcement learning models in our trading agent, if we'd like.\n",
    "\n",
    "![Environment](img/environment.jpeg)\n",
    "\n",
    "Each environment is a fully configurable `gym` environment with highly composable `Portfolio`, `DataFeed`, `ActionScheme`, and `RewardScheme` components.\n",
    "\n",
    "* The `Portfolio` consists of a set of wallets that are each created with an `Exchange` and an `Instrument`.\n",
    "* The `DataFeed` optionally incorporates external data into the the `TradingEnvironment`.\n",
    "* The `ActionScheme` converts the agent's actions into executable trades.\n",
    "* The `RewardScheme` calculates the reward for each time step based on the agent's performance.\n",
    "\n",
    "If it seems a bit complicated now, it's really not. That is all there is to it, now it's just a matter of composing each of these components into a complete environment.\n",
    "\n",
    "When the reset method of a `TradingEnvironment` is called, all of the child components will also be reset. The internal state of the portfolio, data feed, action scheme, and reward scheme will be set back to their default values, ready for the next episode.\n",
    "\n",
    "Let's begin with an example environment. As mentioned before, initializing a `TradingEnvironment` requires an exchange, an action scheme, and a reward scheme, the feature pipeline is optional."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from tensortrade.environments import TradingEnvironment\n",
    "\n",
    "\n",
    "\n",
    "environment = TradingEnvironment(portfolio=portfolio,\n",
    "                                 action_scheme=action_scheme,\n",
    "                                 reward_scheme=reward_scheme,\n",
    "                                 feed=feed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_While the recommended use case is to plug a trading environment into a trading strategy, you can obviously use the trading environment separately, however you'd like._\n",
    "\n",
    "## Exchanges\n",
    "\n",
    "Exchange's determine the tradable instruments within a trading environment. In order to run properly an exchange needs two things:\n",
    "\n",
    "    1.  A mechanism to execute an `Order`\n",
    "    2.  A way to recieve price data for trading pairs.\n",
    "\n",
    "Some examples of combination that can be done in this fashion is:\n",
    "\n",
    "*  (`Simulated`, `Historical`)\n",
    "*  (`Simulated`, `Stochastic`)\n",
    "*  (`Simulated`, `Live`)\n",
    "*  (`Live`, `Live`)\n",
    "\n",
    "The second to last option is called paper trading and the last option is called live trading. A setup like this makes the `Exchange` a highly modular object because the data is separated from the execution system. We will now give an example of each setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulated Execution with Historical Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from tensortrade.exchanges import Exchange\n",
    "from tensortrade.data import Stream\n",
    "\n",
    "def get_historical_data(exchange_name, instrument, timeframe):\n",
    "    path = \"https://www.cryptodatadownload.com/cdd/\"\n",
    "    filename = \"{}_{}USD_{}.csv\".format(exchange, symbol, timeframe)\n",
    "    df = pd.read_csv(path + filename, skiprows=[0])\n",
    "    df = df[::-1]\n",
    "    df = df.drop([\"Symbol\", \"Volume USD\"], axis=1)\n",
    "    df = df.rename({\"Volume {}\".format(symbol): \"Volume\"}, axis=1)\n",
    "    df.columns = [name.lower() for name in df.columns]\n",
    "    df = df.set_index(\"date\")\n",
    "    df.columns = [instrument + \":\" + name for name in df.columns]\n",
    "    return df\n",
    "\n",
    "coinbase_btc = get_historical_data(\"Coinbase\", \"BTC\", \"d\")\n",
    "coinbase_eth = get_historical_data(\"Coinbase\", \"ETH\", \"d\")\n",
    "coinbase_ltc = get_historical_data(\"Coinbase\", \"LTC\", \"d\")\n",
    "\n",
    "exchange = Exchange(\"coinbase\", service=\"simulated\")(\n",
    "    Stream(\"USD-BTC\", list(df2['BTC:close'])),\n",
    "    Stream(\"USD-ETH\", list(df2['ETH:close'])),\n",
    "    Stream(\"USD-LTC\", list(df2['LTC:close']))\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulated Execution with Stochastic Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ================================\n",
    "# Insert code\n",
    "# ================================"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulated Execution with Live Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ================================\n",
    "# Insert code\n",
    "# ================================"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Live Execution with Live Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ================================\n",
    "# Insert code\n",
    "# ================================"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Pipelines\n",
    "\n",
    "Feature pipelines are meant for transforming observations from the environment into meaningful features for an agent to learn from. If a pipeline has been added to a particular exchange, then observations will be passed through the `FeaturePipeline` before being output to the environment. For example, a feature pipeline could normalize all price values, make a time series stationary, add a moving average column, and remove an unnecessary column, all before the observation is returned to the agent.\n",
    "\n",
    "![Pipeline](img/pipeline.jpeg)\n",
    "\n",
    "Feature pipelines can be initialized with an arbitrary number of comma-separated transformers. Each `FeatureTransformer` needs to be initialized with the set of columns to transform, or if nothing is passed, all input columns will be transformed.\n",
    "\n",
    "Each feature transformer has a transform method, which will transform a single observation (a `pandas.DataFrame`) from a larger data set, keeping any necessary state in memory to transform the next frame. For this reason, it is often necessary to reset the `FeatureTransformer` periodically. This is done automatically each time the parent `FeaturePipeline` or `Exchange` is reset.\n",
    "\n",
    "Let's create an example pipeline and add it to our existing exchange."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>close</th>\n",
       "      <th>volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.001349</td>\n",
       "      <td>0.001319</td>\n",
       "      <td>0.001323</td>\n",
       "      <td>0.001295</td>\n",
       "      <td>3.443566e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.001220</td>\n",
       "      <td>0.001211</td>\n",
       "      <td>0.001221</td>\n",
       "      <td>0.001214</td>\n",
       "      <td>-1.976999e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.001079</td>\n",
       "      <td>0.001102</td>\n",
       "      <td>0.001097</td>\n",
       "      <td>0.001116</td>\n",
       "      <td>-5.929822e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.001017</td>\n",
       "      <td>0.001006</td>\n",
       "      <td>0.001016</td>\n",
       "      <td>0.000996</td>\n",
       "      <td>5.926411e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.000974</td>\n",
       "      <td>0.000964</td>\n",
       "      <td>0.000945</td>\n",
       "      <td>0.000948</td>\n",
       "      <td>5.216977e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.000911</td>\n",
       "      <td>0.000925</td>\n",
       "      <td>0.000922</td>\n",
       "      <td>0.000917</td>\n",
       "      <td>2.319959e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.000831</td>\n",
       "      <td>0.000846</td>\n",
       "      <td>0.000842</td>\n",
       "      <td>0.000862</td>\n",
       "      <td>-5.959184e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.000804</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>0.000801</td>\n",
       "      <td>0.000788</td>\n",
       "      <td>-3.857462e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.000735</td>\n",
       "      <td>0.000757</td>\n",
       "      <td>0.000744</td>\n",
       "      <td>0.000767</td>\n",
       "      <td>-3.777224e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.000733</td>\n",
       "      <td>0.000710</td>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000702</td>\n",
       "      <td>1.751349e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.000707</td>\n",
       "      <td>0.000707</td>\n",
       "      <td>0.000733</td>\n",
       "      <td>0.000703</td>\n",
       "      <td>-7.916976e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.000669</td>\n",
       "      <td>0.000678</td>\n",
       "      <td>0.000661</td>\n",
       "      <td>0.000680</td>\n",
       "      <td>4.177735e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.000634</td>\n",
       "      <td>0.000691</td>\n",
       "      <td>0.000615</td>\n",
       "      <td>0.000645</td>\n",
       "      <td>4.024384e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.000633</td>\n",
       "      <td>0.000606</td>\n",
       "      <td>0.000649</td>\n",
       "      <td>0.000612</td>\n",
       "      <td>1.876937e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.000645</td>\n",
       "      <td>0.000622</td>\n",
       "      <td>0.000601</td>\n",
       "      <td>0.000613</td>\n",
       "      <td>2.045871e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.000594</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.000613</td>\n",
       "      <td>0.000626</td>\n",
       "      <td>3.172597e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.000641</td>\n",
       "      <td>0.000627</td>\n",
       "      <td>0.000546</td>\n",
       "      <td>0.000577</td>\n",
       "      <td>2.281644e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.000688</td>\n",
       "      <td>0.000683</td>\n",
       "      <td>0.000621</td>\n",
       "      <td>0.000626</td>\n",
       "      <td>-6.297678e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.000597</td>\n",
       "      <td>0.000611</td>\n",
       "      <td>0.000707</td>\n",
       "      <td>0.000674</td>\n",
       "      <td>-4.302749e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.000564</td>\n",
       "      <td>0.000560</td>\n",
       "      <td>0.000562</td>\n",
       "      <td>0.000584</td>\n",
       "      <td>-1.740256e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.000555</td>\n",
       "      <td>0.000546</td>\n",
       "      <td>0.000548</td>\n",
       "      <td>0.000551</td>\n",
       "      <td>-5.297893e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.000556</td>\n",
       "      <td>0.000551</td>\n",
       "      <td>0.000561</td>\n",
       "      <td>0.000543</td>\n",
       "      <td>-6.324823e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.000541</td>\n",
       "      <td>0.000539</td>\n",
       "      <td>0.000556</td>\n",
       "      <td>0.000545</td>\n",
       "      <td>-5.496024e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.000513</td>\n",
       "      <td>0.000516</td>\n",
       "      <td>0.000533</td>\n",
       "      <td>0.000531</td>\n",
       "      <td>-5.572905e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.000503</td>\n",
       "      <td>0.000503</td>\n",
       "      <td>0.000511</td>\n",
       "      <td>0.000503</td>\n",
       "      <td>-3.001406e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20081</th>\n",
       "      <td>-0.000046</td>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-0.000042</td>\n",
       "      <td>-0.000045</td>\n",
       "      <td>-1.342294e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20082</th>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-0.000053</td>\n",
       "      <td>-0.000052</td>\n",
       "      <td>-0.000046</td>\n",
       "      <td>-4.433407e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20083</th>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-0.000059</td>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-6.511484e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20084</th>\n",
       "      <td>-0.000035</td>\n",
       "      <td>-0.000038</td>\n",
       "      <td>-0.000037</td>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-2.058442e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20085</th>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000037</td>\n",
       "      <td>-0.000034</td>\n",
       "      <td>-0.000035</td>\n",
       "      <td>-6.022800e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20086</th>\n",
       "      <td>-0.000043</td>\n",
       "      <td>-0.000035</td>\n",
       "      <td>-0.000037</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-2.972115e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20087</th>\n",
       "      <td>-0.000014</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000034</td>\n",
       "      <td>-0.000043</td>\n",
       "      <td>-2.787716e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20088</th>\n",
       "      <td>-0.000020</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>-0.000010</td>\n",
       "      <td>-0.000014</td>\n",
       "      <td>6.383715e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20089</th>\n",
       "      <td>-0.000054</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>-0.000044</td>\n",
       "      <td>-0.000020</td>\n",
       "      <td>1.478461e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20090</th>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-0.000050</td>\n",
       "      <td>-0.000044</td>\n",
       "      <td>-0.000054</td>\n",
       "      <td>-2.236390e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20091</th>\n",
       "      <td>-0.000020</td>\n",
       "      <td>-0.000037</td>\n",
       "      <td>-0.000038</td>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-3.184873e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20092</th>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-0.000034</td>\n",
       "      <td>-0.000038</td>\n",
       "      <td>-0.000020</td>\n",
       "      <td>3.398797e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20093</th>\n",
       "      <td>-0.000047</td>\n",
       "      <td>-0.000061</td>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-0.000056</td>\n",
       "      <td>-3.193030e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20094</th>\n",
       "      <td>-0.000036</td>\n",
       "      <td>-0.000045</td>\n",
       "      <td>-0.000039</td>\n",
       "      <td>-0.000047</td>\n",
       "      <td>-1.447067e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20095</th>\n",
       "      <td>-0.000033</td>\n",
       "      <td>-0.000039</td>\n",
       "      <td>-0.000048</td>\n",
       "      <td>-0.000036</td>\n",
       "      <td>3.311241e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20096</th>\n",
       "      <td>-0.000051</td>\n",
       "      <td>-0.000028</td>\n",
       "      <td>-0.000040</td>\n",
       "      <td>-0.000033</td>\n",
       "      <td>3.614130e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20097</th>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000032</td>\n",
       "      <td>-0.000051</td>\n",
       "      <td>4.839105e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20098</th>\n",
       "      <td>-0.000023</td>\n",
       "      <td>-0.000023</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-3.162592e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20099</th>\n",
       "      <td>-0.000034</td>\n",
       "      <td>-0.000034</td>\n",
       "      <td>-0.000038</td>\n",
       "      <td>-0.000023</td>\n",
       "      <td>-1.085959e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20100</th>\n",
       "      <td>-0.000039</td>\n",
       "      <td>-0.000036</td>\n",
       "      <td>-0.000030</td>\n",
       "      <td>-0.000034</td>\n",
       "      <td>6.297284e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20101</th>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-0.000013</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000039</td>\n",
       "      <td>3.229587e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20102</th>\n",
       "      <td>-0.000030</td>\n",
       "      <td>-0.000025</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-3.178813e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20103</th>\n",
       "      <td>-0.000007</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-0.000020</td>\n",
       "      <td>-0.000030</td>\n",
       "      <td>1.210591e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20104</th>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-0.000014</td>\n",
       "      <td>-0.000010</td>\n",
       "      <td>-0.000007</td>\n",
       "      <td>-3.550646e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20105</th>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-0.000010</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-2.211518e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20106</th>\n",
       "      <td>-0.000024</td>\n",
       "      <td>-0.000012</td>\n",
       "      <td>-0.000013</td>\n",
       "      <td>-0.000008</td>\n",
       "      <td>-2.533233e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20107</th>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-0.000018</td>\n",
       "      <td>-0.000016</td>\n",
       "      <td>-0.000024</td>\n",
       "      <td>-1.161787e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20108</th>\n",
       "      <td>0.000011</td>\n",
       "      <td>-0.000005</td>\n",
       "      <td>-0.000014</td>\n",
       "      <td>-0.000026</td>\n",
       "      <td>-1.238842e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20109</th>\n",
       "      <td>0.000012</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>0.000011</td>\n",
       "      <td>-3.995577e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20110</th>\n",
       "      <td>-0.000009</td>\n",
       "      <td>-0.000006</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>-3.144854e+05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20111 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           open      high       low     close         volume\n",
       "0      0.001349  0.001319  0.001323  0.001295  344356.554917\n",
       "1      0.001349  0.001319  0.001323  0.001295  344356.554917\n",
       "2      0.001349  0.001319  0.001323  0.001295  344356.554917\n",
       "3      0.001349  0.001319  0.001323  0.001295  344356.554917\n",
       "4      0.001349  0.001319  0.001323  0.001295  344356.554917\n",
       "...         ...       ...       ...       ...            ...\n",
       "20106 -0.000024 -0.000012 -0.000013 -0.000008 -253323.301085\n",
       "20107 -0.000026 -0.000018 -0.000016 -0.000024 -116178.743729\n",
       "20108  0.000011 -0.000005 -0.000014 -0.000026 -123884.156009\n",
       "20109  0.000012  0.000010  0.000013  0.000011 -399557.736653\n",
       "20110 -0.000009 -0.000006  0.000004  0.000012 -314485.424202\n",
       "\n",
       "[20111 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensortrade.features import FeaturePipeline\n",
    "from tensortrade.features.scalers import MinMaxNormalizer\n",
    "from tensortrade.features.stationarity import FractionalDifference\n",
    "\n",
    "normalize_price = MinMaxNormalizer([\"open\", \"high\", \"low\", \"close\"])\n",
    "difference_all = FractionalDifference(difference_order=0.6)\n",
    "\n",
    "feature_pipeline = FeaturePipeline(steps=[normalize_price, difference_all])\n",
    "\n",
    "exchange.feature_pipeline = feature_pipeline\n",
    "\n",
    "exchange.data_frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This feature pipeline normalizes the price values between 0 and 1, before making the entire time series stationary by fractionally differencing consecutive values._\n",
    "\n",
    "## Action Strategies\n",
    "\n",
    "Action schemes define the action space of the environment and convert an agent's actions into executable trades. For example, if we were using a discrete action space of 3 actions (0 = `hold`, 1 = `buy 100%`, 2 = `sell 100%`), our learning agent does not need to know that returning an action of 1 is equivalent to buying an instrument. Rather, our agent needs to know the reward for returning an action of 1 in specific circumstances, and can leave the implementation details of converting actions to trades to the `ActionScheme`.\n",
    "\n",
    "Each action scheme has a get_trade method, which will transform the agent's specified action into an executable `Trade`. It is often necessary to store additional state within the scheme, for example to keep track of the currently traded position. This state should be reset each time the action scheme's reset method is called, which is done automatically when the parent `TradingEnvironment` is reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensortrade.actions import DiscreteActions\n",
    "\n",
    "action_scheme = DiscreteActions(n_actions=20, instrument='BTC')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This discrete action scheme uses 20 discrete actions, which equates to 4 discrete sizes for each of the 5 trade types (market buy/sell, limit buy/sell, and hold). E.g. [0,5,10,15]=hold, 1=market buy 25%, 2=market sell 25%, 3=limit buy 25%, 4=limit sell 25%, 6=market buy 50%, 7=market sell 50%, etc…_\n",
    "\n",
    "## Reward Strategies\n",
    "\n",
    "Reward schemes receive the trade taken at each time step and return a float, corresponding to the benefit of that specific action. For example, if the action taken this step was a sell that resulted in positive profits, our `RewardScheme` could return a positive number to encourage more trades like this. On the other hand, if the action was a sell that resulted in a loss, the scheme could return a negative reward to teach the agent not to make similar actions in the future.\n",
    "\n",
    "A version of this example algorithm is implemented in `SimpleProfit`, however more complex schemes can obviously be used instead.\n",
    "\n",
    "Each reward scheme has a get_reward method, which takes in the trade executed at each time step and returns a float corresponding to the value of that action. As with action schemes, it is often necessary to store additional state within a reward scheme for various reasons. This state should be reset each time the reward scheme's reset method is called, which is done automatically when the parent `TradingEnvironment` is reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensortrade.rewards import SimpleProfit\n",
    "\n",
    "reward_scheme = SimpleProfit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_The simple profit strategy returns a reward of -1 for not holding a trade, 1 for holding a trade, 2 for purchasing an instrument, and a value corresponding to the (positive/negative) profit earned by a trade if an instrument was sold._\n",
    "\n",
    "## Learning Agents\n",
    "\n",
    "Up until this point, we haven't seen the \"deep\" part of the deep reinforcement learning framework. This is where learning agents come in. Learning agents are where the math (read: magic) happens.\n",
    "\n",
    "![Agent](img/agents.jpeg)\n",
    "\n",
    "At each time step, the agent takes the observation from the environment as input, runs it through its underlying model (a neural network most of the time), and outputs the action to take. For example, the observation might be the previous open, high, low, and close price from the exchange. The learning model would take these values as input and output a value corresponding to the action to take, such as buy, sell, or hold.\n",
    "\n",
    "It is important to remember the learning model has no intuition of the prices or trades being represented by these values. Rather, the model is simply learning which values to output for specific input values or sequences of input values, to earn the highest reward.\n",
    "\n",
    "## Stable Baselines\n",
    "\n",
    "In this example, we will be using the Stable Baselines library to provide learning agents to our trading strategy, however, the TensorTrade framework is compatible with many reinforcement learning libraries such as Tensorforce, Ray's RLLib, OpenAI's Baselines, Intel's Coach, or anything from the TensorFlow line such as TF Agents.\n",
    "\n",
    "It is possible that custom TensorTrade learning agents will be added to this framework in the future, though it will always be a goal of the framework to be interoperable with as many existing reinforcement learning libraries as possible, since there is so much concurrent growth in the space.\n",
    "\n",
    "But for now, Stable Baselines is simple and powerful enough for our needs."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from stable_baselines.common.policies import MlpLnLstmPolicy\n",
    "from stable_baselines import PPO2\n",
    "\n",
    "model = PPO2\n",
    "policy = MlpLnLstmPolicy\n",
    "params = { \"learning_rate\": 1e-5, 'nminibatches': 1 }\n",
    "\n",
    "agent = model(policy, environment, **params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Note: Stable Baselines is not required to use TensorTrade though it is required for this tutorial. This example uses a GPU-enabled Proximal Policy Optimization model with a layer-normalized LSTM perceptron network. If you would like to know more about Stable Baselines, you can view the [Documentation](https://stable-baselines.readthedocs.io/en/master/)._\n",
    "\n",
    "## Tensorforce\n",
    "\n",
    "I will also quickly cover the Tensorforce library to show how simple it is to switch between reinforcement learning frameworks."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from tensorforce.agents import Agent\n",
    "\n",
    "network_spec = [\n",
    "    dict(type='dense', size=128, activation=\"tanh\"),\n",
    "    dict(type='dense', size=64, activation=\"tanh\"),\n",
    "    dict(type='dense', size=32, activation=\"tanh\")\n",
    "]\n",
    "\n",
    "agent_spec = {\n",
    "    \"type\": \"ppo\",\n",
    "    \"learning_rate\": 1e-4,\n",
    "    \"discount\": 0.99,\n",
    "    \"likelihood_ratio_clipping\": 0.2,\n",
    "    \"estimate_terminal\": False,\n",
    "    \"max_episode_timesteps\": 2000,\n",
    "    \"network\": network_spec,\n",
    "    \"batch_size\": 10,\n",
    "    \"update_frequency\": \"never\"\n",
    "}\n",
    "\n",
    "agent = Agent.create(agent=agent_spec, environment=environment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_If you would like to know more about Tensorforce agents, you can view the [Documentation](https://tensorforce.readthedocs.io/en/0.4.4)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trading Strategy\n",
    "\n",
    "A `TradingStrategy` consists of a learning agent and one or more trading environments to tune, train, and evaluate on. If only one environment is provided, it will be used for tuning, training, and evaluating. Otherwise, a separate environment may be provided at each step."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from tensortrade.strategies import TensorforceTradingStrategy,\n",
    "                                   StableBaselinesTradingStrategy\n",
    "    \n",
    "a_strategy = TensorforceTradingStrategy(environment=environment,\n",
    "                                        agent_spec=agent_spec,\n",
    "                                        network_spec=network_spec)\n",
    "\n",
    "b_strategy = StableBaselinesTradingStrategy(environment=environment,\n",
    "                                            model=PPO2,\n",
    "                                            policy=MlpLnLSTMPolicy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Don't worry if you don't understand the strategy initialization just yet, it will be explained in more detail later._\n",
    "\n",
    "# Putting it All Together\n",
    "\n",
    "Now that we know about each component that makes up a `TradingStrategy`, let's build and evaluate one.\n",
    "\n",
    "![Tuning](img/tuning.jpeg)\n",
    "\n",
    "For a quick recap, a `TradingStrategy` is made up of a `TradingEnvironment` and a learning agent. A `TradingEnvironment` is a gym environment that takes an `Exchange`, an `ActionScheme`, a `RewardScheme`, and an optional `FeaturePipeline`, and returns observations and rewards that the learning agent can be trained and evaluated on.\n",
    "\n",
    "## Creating an Environment\n",
    "\n",
    "The first step is to create a `TradingEnvironment` using the components outlined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensortrade.environments import TradingEnvironment\n",
    "\n",
    "environment = TradingEnvironment(exchange=exchange,\n",
    "                                 feature_pipeline=feature_pipeline,\n",
    "                                 action_scheme=action_scheme,\n",
    "                                 reward_scheme=reward_scheme)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple enough, now `environment` is a `gym` environment that can be used by any compatible trading strategy or learning agent.\n",
    "\n",
    "## Defining the Agent\n",
    "\n",
    "Now that the environment is set up, it’s time to create our learning agent. Again, we will be using Stable Baselines for this, but feel free to drop in any other reinforcement learning agent here.\n",
    "\n",
    "Since we are using `StableBaselinesTradingStrategy`, all we need to do is provide a model type and a policy type for the underlying neural network to be trained. For this example, we will be using a simple proximal policy optimization (PPO) model and a layer-normalized LSTM policy network.\n",
    "\n",
    "For more examples of model and policy specifications, see the [Stable Baselines Documentation](http://stable-baselines.readthedocs.io/en/master/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines.common.policies import MlpLnLstmPolicy\n",
    "from stable_baselines import PPO2\n",
    "\n",
    "model = PPO2\n",
    "policy = MlpLnLstmPolicy\n",
    "params = { \"learning_rate\": 1e-5, 'nminibatches': 1 }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a Strategy\n",
    "\n",
    "Creating our trading strategy is as simple as plugging in our agent and the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensortrade.strategies import StableBaselinesTradingStrategy\n",
    "\n",
    "strategy = StableBaselinesTradingStrategy(environment=environment,\n",
    "                                          model=model,\n",
    "                                          policy=policy,\n",
    "                                          model_kwargs=params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then to train the strategy (i.e. train the agent on the current environment), all we need to do is call `strategy.run()` with the total number of steps or episodes you’d like to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished running strategy.\n",
      "Total episodes: 0 (10000 timesteps).\n",
      "Average reward: -0.38326355742842805.\n"
     ]
    }
   ],
   "source": [
    "performance = strategy.run(steps=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x172bad9d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhU1Z3/8fe3qnpvoFkaZJUWUESJSlrQEOMGikvE8TFqzBPRGP1lTEYzZmJ0EmPiHicTlxmTDFGjzm8SNepvJK5BQY07jSuyNovSCDRL0zQNvVTX+f1xTzfddIPYS92uqs/refqpc8+9Vf2to/Dhbueacw4REclskbALEBGR8CkMREREYSAiIgoDERFBYSAiIkAs7AI6a9CgQW706NFhlyEikjIWLly42TlX3NG6lA2D0aNHU1ZWFnYZIiIpw8w+2ds6HSYSERGFgYiIKAxERASFgYiIoDAQERH2IwzM7AEzqzSzRa36BpjZXDNb4V/7+34zs3vMrNzMPjSzSa3eM8tvv8LMZrXq/7KZfeTfc4+ZWXd/SRER2bf92TN4EJixR9+1wEvOuXHAS34Z4DRgnP+5HPgdBOEB3ABMASYDNzQHiN/mslbv2/N3iYhID/vcMHDOvQps3aN7JvCQbz8EnN2q/2EXeAsoMrOhwKnAXOfcVudcFTAXmOHX9XXOveWCubQfbvVZ+1TfmNifzUREZD909pzBEOfcet/eAAzx7eHA2lbbVfi+ffVXdNDfITO73MzKzKyscmtVJ0sXEZE9dfkEsv8XfVKekOOcm+2cK3XOlTZFc5PxK0VEMkJnw2CjP8SDf630/euAka22G+H79tU/ooP+z7WjPt6pwkVEpL3OhsEcoPmKoFnAU636L/JXFR0DVPvDSS8Ap5hZf3/i+BTgBb9uu5kd468iuqjVZ+2TAxIJPbJTRKQ77M+lpX8G3gQOMbMKM7sUuB2YbmYrgGl+GeBZYBVQDvwBuALAObcVuAlY4H9u9H34be7z71kJPLe/xc/54LP93VRERPbBgkP+qSdn6Dg3dNZdrLn9jLBLERFJCWa20DlX2tG6lL8DWYeKRES6LuXDYOmGmrBLEBFJeSkbBsWFOQCcfs/fQ65ERCT1pWwYDOmr+wxERLpLyoaBGQztFwSCzhuIiHRNyoYBwLlfDu5Xu/ulFbxRvjnkakREUldKh8GFU0YBQRhceN/bIVcjIpK6UjoMhvbLa7NcvbMxpEpERFJbSofBnt5dq5lMRUQ6I+XD4PozJ7S0K6p2hViJiEjqSvkwuPSrJfzyrMMA+HhddcjViIikppQPA4BZXxkNwCML1u57QxER6VBahEFra7fuDLsEEZGUkzZh8OjlxwDw/tptIVciIpJ60iYMJh3YH4CfP7Uo5EpERFJP2oRBVjT4KlU7G/mwQnsHIiJfRNqEAcAVJ4wB4Kz/fJ1UfWiPiEgY0ioMrjx5XEt73tLKECsREUktaRUGuVlRbvmHwwG49KGykKsREUkdaRUGAOeVjgy7BBGRlJN2YZAVjTCifzCB3SdbakOuRkQkNaRdGAD8+NRDAPhsW13IlYiIpIa0DIPxB/QFYOWmHSFXIiKSGtIyDIb0zQHg96+sDLkSEZHUkJZhUJSfDcDhw/qFXImISGpIyzAAGDe4ELOwqxARSQ1pGwb5OTFqG5rCLkNEJCWkbRgUZEeprY+HXYaISEpI2zDIz44pDERE9lPahkHfvBjbdzWGXYaISEroUhiY2T+b2cdmtsjM/mxmuWZWYmZvm1m5mT1qZtl+2xy/XO7Xj271Odf5/mVmdmrXvlJgcJ9cNu9o0OylIiL7odNhYGbDgSuBUufc4UAUuAD4FXCnc24sUAVc6t9yKVDl++/022FmE/z7DgNmAL81s2hn62rWNy9GQ1OCe+eXd/WjRETSXlcPE8WAPDOLAfnAeuAk4HG//iHgbN+e6Zfx6082M/P9jzjn6p1zq4FyYHIX6+LMicMA+PXflnf1o0RE0l6nw8A5tw74NfApQQhUAwuBbc655jO3FcBw3x4OrPXvjfvtB7bu7+A9nTZqYH5LO96U6OrHiYikta4cJupP8K/6EmAYUEBwmKfHmNnlZlZmZmWbNm363O3P99NZr6/WhHUiIvvSlcNE04DVzrlNzrlG4ElgKlDkDxsBjADW+fY6YCSAX98P2NK6v4P3tOGcm+2cK3XOlRYXF39ugdMnDAHgqfc7/DgREfG6EgafAseYWb4/9n8ysBiYD5zrt5kFPOXbc/wyfv08F1zqMwe4wF9tVAKMA97pQl0tDuiXC+i8gYjI54l9/iYdc869bWaPA+8CceA9YDbwDPCImd3s++73b7kf+G8zKwe2ElxBhHPuYzN7jCBI4sD3nXPdMo/E4cN3T1TnnMM0WZGISIc6HQYAzrkbgBv26F5FB1cDOefqgG/s5XNuAW7pSi17892vlnDfa6upbWiiMKdLX1dEJG2l7R3IzcYOLgTQ3cgiIvuQ9mHQNy8LgJo6zVMkIrI36R8GuUEYbK/TnoGIyN6kfRj0yQ3OE/zuZT0CU0Rkb9I+DJoPE81bWkmN9g5ERDqU9mEwoCC7pf34wooQKxER6b3SPgz65u6+nPSXf10cYiUiIr1X2oeBmXH3BUe2LO9s0FVFIiJ7SvswAJh55HDOmRRMhDrh5y9w2cNlIVckItK7ZEQYAFzylZKW9tzFG0OsRESk98mYMJg4oh/vXj+dia3mKxIRkUDGhAEEVxZNOzSY1nreUu0diIg0y6gwABhQENx38J0Hy/juQ2XMX1oZckUiIuHLuDA4alT/lvaLSzZyyYMLCB6rICKSuTIuDA4d2rdd367Gbnl8gohIysq4MIhGjOU3n8a/nj6+pW9lZW2IFYmIhC/jwgAgOxbhsuMO4nvHjwFgc219yBWJiIQrI8MAgjuTvzVlFACbajoOg3hTgtHXPsPVj76fzNJERJIuY8MAoLhPDgCV2+s6XL90Qw0AT763Lmk1iYiEIaPDIDcrCsCv/7a8wyuKXlm+qaVdevNcmhK66khE0lNGh0FrW2sb2vWNGpDf0t68o4HHF65NZkkiIkmT8WFw/ZkTANjZ0P7y0sQeews//X+LklKTiEiyZXwYDO2XC8Br5ZvZvKPtieR4UxAGT//TV/ne8WOIJxzfvv/tpNcoItLTMj4MCnKCh99c9+RHTL19Xpt1TX7PoF9eFmcdMQyAv6/YzEcV1cktUkSkh2V8GJQMLGhp18cTbU4kN58wjkWN8Qf0aemfee9ryStQRCQJMj4MRg7Ia7P8zuqt1NbHWV+9qyUMomZEIsaqW08HIOHgzZVbkl6riEhPyfgwMLM2y+fPfouJv3iBY2+b13L/QTQSbBOJGFeeNBaAb/7hreQWKiLSgzI+DDrSfDvBPfPKgd1hAHDVtIPDKElEpEcpDIDvnziGX3x9Aj+ZMb7D9a3DIBoxxhQH5xmefLciKfWJiPQ0hQHw41PHc/HUEv7xhDEdrs+Kth2m0w4fCsDVj33Q47WJiCSDwmAPZ0wc2q6vedqKZleePC5Z5YiIJEWXwsDMiszscTNbamZLzOxYMxtgZnPNbIV/7e+3NTO7x8zKzexDM5vU6nNm+e1XmNmsrn6prvjm5FEt7WU3z2DN7We02yY7tnvYqnc2JqUuEZGe1NU9g7uB551z44EjgCXAtcBLzrlxwEt+GeA0YJz/uRz4HYCZDQBuAKYAk4EbmgMkDF8dN4irpx/Mr79xBDmx6F63u2bGIQD89cPPklWaiEiP6XQYmFk/4GvA/QDOuQbn3DZgJvCQ3+wh4Gzfngk87AJvAUVmNhQ4FZjrnNvqnKsC5gIzOltXd7jy5HGc++UR+9zme18Lzi+8uGRjMkoSEelRXdkzKAE2AX80s/fM7D4zKwCGOOfW+202AEN8ezjQetrPCt+3t/52zOxyMyszs7JNmzZ1tEnSRCLGOZOG8/KyTcxbqkAQkdTWlTCIAZOA3znnjgJq2X1ICAAXzO3QbQ8BcM7Nds6VOudKi4uLu+tjO+2npx8KwHceLCPelAi5GhGRzutKGFQAFc655mk8HycIh43+8A/+tdKvXweMbPX+Eb5vb/293sDCnJb2b+YuD7ESEZGu6XQYOOc2AGvN7BDfdTKwGJgDNF8RNAt4yrfnABf5q4qOAar94aQXgFPMrL8/cXyK70sJz155HAC/fXml9g5EJGXFuvj+fwL+x8yygVXAJQQB85iZXQp8Apznt30WOB0oB3b6bXHObTWzm4AFfrsbnXNbu1hX0kwY1relfd5/vcmTV0wNsRoRkc6xjp79mwpKS0tdWVlZ2GUAsGJjDdPvfBWgw/sSRER6AzNb6Jwr7Wid7kDuBuOG9OEwv4fwizkfh1yNiMgXpzDoJnedfyQAD76xhmUbakKuRkTki1EYdJNxQ/pw2zkTATj1rldDrkZE5ItRGHSjb7S6a/myh3vH+QwRkf2hMOhGsWiEey8M5t+bu3gjry4P9y5pEZH9pTDoZmd8aSjTJwQzcFz0wDstz1EWEenNFAY94A8X7b5y64LZb4ZYiYjI/lEY9JB3/vVkABasqeK255Zw14vLadQdyiLSSykMesjgvrlMHTsQgP96ZRV3vbiCJ9+tYPS1z/D3FTqXICK9i8KgB519ZNuZuH/yxEcA/PCR98MoR0RkrxQGPejcL4/gvy+dzJjigjb9W2obSNVpQEQkPSkMepCZcdy4Yv73+1M54ZC2z1845GfPs756V0iViYi0pTBIgj65Wfz2W5Pa9DU0JTj2tnk6qSwivYLCIEnys3fPFl4yaPdho9fKN4dRjohIG119noF8AatvO5311XUMKMhm/PXPAxA1C7kqERHtGSSVmTGsKI/crCgH+ZPKa6t2hlyViIjCIDQv/vPxHFRcwDMfrg+7FBERhUFYIhFj3OBCtuxoCLsUERGFQZiK8rKp2qkwEJHwKQxCVFSQxbZdjboBTURCpzAI0bB+eTTEE2zYXhd2KSKS4RQGITp4SB8APTNZREKnMAjRhKF9Abj5mSUtfY1NCXY2xPlg7TYa4ro7WUSSQzedhahffhZHj+7PgjVVvLh4I9MmDOHiP77D6+VbWrZZcctpZEWV2SLSs/S3TMjuPP9I+uVl8d2Hy9i4va5NEACs2VwbUmUikkkUBiEb0T+fuy44EoApt77EiX5200mjigDYXhcPrTYRyRwKg17ghIN3T289f1nwFLQbZx4OwKvL9VQ0Eel5CoNewMxYetOMNn3NJ5fvfmkFdY1NYZQlIhlEYdBL5GZFue2ciQA8/r1jiUSM/OwoAD/407thliYiGUBh0It8c/Io1tx+BqWjBwDw3FXHAfDikkrtHYhIj1IY9GIHDixg5pHDAHhr1RYSCcfWWs1lJCLdr8thYGZRM3vPzJ72yyVm9raZlZvZo2aW7ftz/HK5Xz+61Wdc5/uXmdmpXa0pndx2zkSyoxFeXb6ZB15fzaSb5nLsbS+FXZaIpJnu2DO4CljSavlXwJ3OubFAFXCp778UqPL9d/rtMLMJwAXAYcAM4LdmFu2GutJCfnaMSQcWUfbJVipr6gFYX13Hat1/ICLdqEthYGYjgDOA+/yyAScBj/tNHgLO9u2Zfhm//mS//UzgEedcvXNuNVAOTO5KXelm6phBfFhRzbptu1r6rn7s/RArEpF009U9g7uAa4DmSXQGAtucc813SlUAw317OLAWwK+v9tu39HfwnjbM7HIzKzOzsk2bMuf6+/OPHgnQ8lS0kkEFYZYjImmo02FgZmcClc65hd1Yzz4552Y750qdc6XFxcWf/4Y0Mbhvbpvl4sIc3vt0W0jViEg66sqewVTgLDNbAzxCcHjobqDIzJonwBsBrPPtdcBIAL++H7CldX8H7xEvJxb8pxpUmM3OxmDH69755WGWJCJppNNh4Jy7zjk3wjk3muAE8Dzn3LeA+cC5frNZwFO+Pccv49fPc8EjvuYAF/irjUqAccA7na0rXTVfYnrxV0Zz1/lHAbCyckeYJYlIGumJ+wx+AlxtZuUE5wTu9/33AwN9/9XAtQDOuY+Bx4DFwPPA951zusNqDz+ZMZ5phw7mwikHMnZwIWOKC3jyvXUcfcuL7KhvP5ndlFtf5Jzfvh5CpSKSirrleQbOuZeBl317FR1cDeScqwO+sZf33wLc0h21pKuBhTncN+voluW6xuCc/aaaev709id8Z2oJd7ywjOUba7jr/CPZuL2ejdvrWbt1JyMH5IdVtoikCEvVh7GXlpa6srKysMsIzQOvrebGpxfv17blt5xGTA/IEcl4ZrbQOVfa0Tr9DZGiLv7KaO4490vt+scOLmzf99PnSNXQF5HkUBikqEjEOK90JBccPbJN/5++O4X3fz6dnFiEP102paV/0k1zk12iiKQQHSZKA00Jxy//+jHrqnZx36xSghu7A845Sq57FoA1t58RVoki0gvs6zBRt5xAlnBFI9byZLQ9mQXPRdjZ0MT9r63mnKOG078gO8kVikhvp8NEGeBbU0YBcNPTi/nho5rTSETaUxhkgOkTDmhpv6JnKotIBxQGGWByyYA2y796fmlIlYhIb6UwyBB/v+bElvbvXl4ZYiUi0hspDDLEyAH5rLn9DKYdOpjiPjlhlyMivYzCIMMcPKQPm2rque/vq8IuRUR6EYVBhpl5ZPDcoJufWcINTy0KuRoR6S0UBhnmkAP6kJ8dPGL6oTc/obKmLuSKRKQ3UBhkoMU3zuDBS4IZUK9+9IOQqxGR3kBhkKEOGhRMaPda+WY+27Yr5GpEJGwKgww1amA+p08MbkZ79qP1IVcjImFTGGSwey+cRHY0wjrtGYhkPIVBBjMz8nOi/PH1NcxfWhl2OSISIoVBhrvJz3b6jA4ViWQ0hUGG+/oRwxh/QB/WVelQkUgmUxgIR40q4s1VW3hVM5qKZCyFgfDP0w4G4M/vfOqfjPYM//IX3X8gkkkUBsLgvrlMGlXEc4s2cNRNc3EOHl9YQao+ElVEvjiFgQDwL6ceAsC2nY0tfSXXPUttfTyskkQkiRQGAsBXxgziz5cd067/kj8uaGkv21CjvQWRNGWp+oe7tLTUlZWVhV1G2vmoopp5SyspzI1x09OL260/4ZBiHrxkcgiViUhXmdlC51xpR+u0ZyBtTBzRj6umjePSr5Z0uP7lZZsYfe0zzFu6McmViUhPUhjIfjmvdASnHX5Ay/K//215iNWISHeLhV2A9F4rbz2dusYmCnJ2/2+yva6RL/3ib3z82XbKK2u49dmlzP72l4lGjMqaeob0zQ2xYhHpLIWB7FU0Ym2CAKBvblZLe9pvXgXgsofLmL8suGHtj5cczWsrNlOUl8U/nTwuecWKSJd0+jCRmY00s/lmttjMPjazq3z/ADOba2Yr/Gt/329mdo+ZlZvZh2Y2qdVnzfLbrzCzWV3/WtKTFv5sWpvl5iCA4Oqj+19bzb/PXU555Y5klyYindSVcwZx4EfOuQnAMcD3zWwCcC3wknNuHPCSXwY4DRjnfy4HfgdBeAA3AFOAycANzQEivdPAwpz92m7ab17hndVbe7gaEekOnQ4D59x659y7vl0DLAGGAzOBh/xmDwFn+/ZM4GEXeAsoMrOhwKnAXOfcVudcFTAXmNHZuiQ5rjhhTJvlP15yNO9dP73ddo+882myShKRLuiWcwZmNho4CngbGOKca54PeQMwxLeHA2tbva3C9+2tX3qxa2aM55oZ46mqbaAxkWBwn+DE8Us/Op6VlTuo2tnAT574iHnLKnHOYWYhVywi+9LlS0vNrBB4Avihc25763UuuKOt2+5qM7PLzazMzMo2bdIMm71B/4LsliAAGFNcyCmHHcD5R4/iByeOZdvORm5+ZgkfVVQz+tpneOr9de0+442Vm3XfgkjIuhQGZpZFEAT/45x70ndv9Id/8K/Nj9BaB4xs9fYRvm9v/e0452Y750qdc6XFxcVdKV2S4MIpowC4/7XVfP0/XwPgqkfe54mFFQBUbq9jzeZaLvzD23znwTISidS8G14kHXTlaiID7geWOOd+02rVHKD5iqBZwFOt+i/yVxUdA1T7w0kvAKeYWX9/4vgU3ycpblhRHg9c3P7O9x/95QM2VNcx+daXOOHXL7f0P7JgbbttRSQ5urJnMBX4NnCSmb3vf04Hbgemm9kKYJpfBngWWAWUA38ArgBwzm0FbgIW+J8bfZ+kgZPGD+GVH5/Qrv+OF5a267vx6Y9piCeSUJWI7EkT1UlSvLJ8E6MG5PPCxxu4/bn2QdDsO1NL+PnXJySxMpHMoYnqJHTHH1xMyaACqmobWvoGFWa3tJunz37g9dUsWled9PpEMp3CQJLqrCOHtbTLfrb7voRxQwq555tHAXDmf7zGms21Sa9NJJMpDCSpDhvWjzvO/RJvXHtSm/6C7BhnHTGMovxg7qMTfv0yjy7QDWsiyaIwkKQ7r3Qkw4rygODw0KVfLSEvOwrA+z8/hbysoP3Lv7Z/uI6I9AyFgYTq2DEDuf7MtieMn//hcQAc0E/TYYski8JAep0DBxZwfulIVm2q5d755WGXI5IRFAbSK33/xLEA/NsLy7jlmcVsr2vUPQgiPUj3GUivVdfYxI/+8gHPfLi+pW9ov1y+OXkUV+rBOSJfmO4zkJSUmxXl7vOP5P8cf1BL3/rqOn4zdzmrNunBOSLdSXsGklIefH01v/jrYgpzYjx31XGMHJAfdkkiKUN7BpI2Lp5awp8vO4Yd9XH+4bdvUB9vCrskkbSgMJCUc+yYgVwz4xA276jnkJ89z5h/fZYVG2tI1b1ckd6gW550JpJsV5wwlsF9crn+fxexq7GJ6Xe+CsDwojx+cNJYzi8dSU1dnI/XVzOlZCDRiJ60JrIvOmcgKe/WZ5cw+9VVe10/akA+/3nhURTmxDiouDCJlYn0Lvs6Z6AwkLRQ19hELGL837c+4RetprEYWJDNllYzpZYMKiAnFuGnZxzKceP0tDzJLDqBLGkvNytKLBrh4qklLLlxBsP8/QgLr5/O944f07Ld6s21LN1Qw7fvf4f/fa/Dp6uKZCTtGUhGSCQcZvDup9sor6zhJ098RL+8LH551mGcNvEAcmLRsEsU6XE6TCSyh0fe+ZRrn/wIgOI+OZz75REcf3Axxxw0MOTKRHqOwkCkA9U7G3msbC23PLukTf+JhxRzx7lHUNwnJ6TKRHqGzhmIdKBffhaXfe0gnrvqOM4rHcGhQ/sCMH/ZJs79/RvU1DVy1I1/49Q7XyWRSM1/NInsL+0ZiLTyxsrNXPiHtztc95fvHcvE4f3IzdL5BUlN2jMQ2U9fGTOIV398Iiccsvuy04nD+wHwjd+/yfjrn+drd8xnV4OmwZD0oj0Dkb2o3F5HTixKv/wsKqp2csHst6io2gUE9ys8eMnRHDiwIOQqRfafTiCLdBPnHN/4/Zu8t3YbTQlHTizCOZNGMHZwIR9VbOMfJgVXJYn0RgoDkW62cXsdD7y+mgdfX0P9Hk9gO27cII4YUcTkkgF8aUQ/ivKzQ6pSpC2FgUgPqmtsorxyB6s21/LB2m28vKyST7bsJO6vQBpUmM2hQ/uyobqOqWMHcVBxAROH92P8AX3Jy9bJaEkehYFIku1siPPmyi2s2lTLso01vLBoAzX18XbbHTgwnyklAxh/QF8mlwxg7OBCXa0kPUZhIBKyeFOCmro4RflZfFZdx0cV1Sxev533125j0bpqtvrJ9LKixpjiQoYV5bGltoF+eVkcPqwvBxUXMrhPDkP65jKkbw798rIw07Tc8sXsKwz0PAORJIhFI/QvCM4dDC/KY3hRHjMOPwAITkqv2bKTt1dtYcn67VRU7eKz6jpWb95BXWOCN8o3txxyahaNGIU5MaYdOoQfThunx39KlykMREJmZpQMKqBkUNvLVJ1zmBl1jU1s3F5HZU09G7fXsX5bHZU1dSzdUMMT71bwxLsVHD26P8V9chhQkE1eVpQ+uVkM7pND/4Js+udnM7hPDgMKs+mTE9MehXRIYSDSSzX/pZ2bFeXAgQUd3tPw6IJPeb18Cxu317FsQw1baxuoa0ywq7Hjm+KiEaN/fjaDCrMZUJDNwMIcBhZkM7Agm6L8LGLRCDmxCAU5MQqyYxTkRCnMiZGfE6PQL8eiu+9VbUo4PUUuTfSaMDCzGcDdQBS4zzl3e8glifR65x89ivOPHtWuvz7exOYdDVTVNlC1s4GN2+tb2lU7G9iyo4EttQ0sWlfN5h311NS1P7m9N9mxCIU5MRqbEuxqaGJI31zysqNkRyNkxSKMGpDPwIJs8rOjPlSi5GZFycmKkJcVJS87RmFOsC43FiUrFiFqRjRixCJGNOpfI4YRtCMKnB7XK8LAzKLAvcB0oAJYYGZznHOL9/1OEelITizacm5if9THm6je1UhTwlHXmKC2Ps7OhiZq6+PsqI+zsyHOjvomdtbH2dEQp7Y+Tk1dnPrGBAU5MXY1xmmIO3Y2xHn3kypq6hqpbWiiqZsm+ItGjKyokRWNkB2NkB2LBO1Wr9lR270cjRCLGs6Bc5BwwfMsYpGgPxbZ/Z5IxFrCqLkdiewOpObfu+fvzooaWbGgr3k5OxYJPseMiAV7dxEzDIJXg0gkWNfc37wXZwTro/73RnxNQT100Ne9AdkrwgCYDJQ751YBmNkjwExAYSCSBDmxKIP7dO8lrc456uNBsNTHE9THgz2JIFji1NY3UR9vIt7kiCccTYkETYnmdvDqnKMpAY1NCRqbgs9objfEEzQ2uZa+hniC+sbgqq2GeIJ4wrX5S9g5iCeC/sZ4gsaEoyGeIJFwNLngdyZaXrt1KHpMm7DwAdE6LHaHXBCE+9JbwmA4sLbVcgUwZc+NzOxy4HKAUaPa7xqLSO9hZuRmRVPyvgnnQ6E5JBriCRqagvBp9O2GlmByu8MonsC5IEwSLggX19LGL7deD/l+fBy+z//eRKK5BtoGVuv1PiybQ6xtoDX37V7/8j6+c28Jg/3inJsNzIbgPoOQyxGRNGVmweEkv5wuM4r8x4V7X9dbprBeB4xstTzC94mISBL0ljBYAIwzsxIzywYuAOaEXJOISMboFYeJnHNxM/sB8ALBpaUPOOc+DrksEZGM0SvCAMA59yzwbNh1iIhkoqDxAe0AAAQvSURBVN5ymEhEREKkMBAREYWBiIgoDEREhBR+uI2Z1QDLwq6jFxkEbA67iF5GY9KWxqO9TBuTA51zxR2t6DVXE3XCsr09sScTmVmZxqMtjUlbGo/2NCa76TCRiIgoDEREJLXDYHbYBfQyGo/2NCZtaTza05h4KXsCWUREuk8q7xmIiEg3URiIiEjqhYGZzTCzZWZWbmbXhl1PTzKzB8ys0swWteobYGZzzWyFf+3v+83M7vHj8qGZTWr1nll++xVmNiuM79IdzGykmc03s8Vm9rGZXeX7M3JMzCzXzN4xsw/8ePzS95eY2dv+ez/qp4XHzHL8crlfP7rVZ13n+5eZ2anhfKPuY2ZRM3vPzJ72yxk/Jp/L+cewpcIPwfTWK4GDgGzgA2BC2HX14Pf9GjAJWNSq7w7gWt++FviVb58OPAcYcAzwtu8fAKzyr/19u3/Y362T4zEUmOTbfYDlwIRMHRP/vQp9Owt423/Px4ALfP/vgX/07SuA3/v2BcCjvj3B/1nKAUr8n7Fo2N+vi2NzNfAn4Gm/nPFj8nk/qbZnMBkod86tcs41AI8AM0Ouqcc4514Ftu7RPRN4yLcfAs5u1f+wC7wFFJnZUOBUYK5zbqtzrgqYC8zo+eq7n3NuvXPuXd+uAZYQPD87I8fEf68dfjHL/zjgJOBx37/neDSP0+PAyWZmvv8R51y9c241UE7wZy0lmdkI4AzgPr9sZPiY7I9UC4PhwNpWyxW+L5MMcc6t9+0NwBDf3tvYpOWY+d35owj+NZyxY+IPh7wPVBKE2kpgm3Mu7jdp/d1avrdfXw0MJI3Gw7sLuAZI+OWBaEw+V6qFgbTigv3ZjLs22MwKgSeAHzrntrdel2lj4pxrcs4dSfDc8MnA+JBLCpWZnQlUOucWhl1Lqkm1MFgHjGy1PML3ZZKN/lAH/rXS9+9tbNJqzMwsiyAI/sc596TvzugxAXDObQPmA8cSHA5rnnes9Xdr+d5+fT9gC+k1HlOBs8xsDcFh5JOAu8nsMdkvqRYGC4Bx/sqAbIITPnNCrinZ5gDNV7/MAp5q1X+Rv4LmGKDaHzp5ATjFzPr7q2xO8X0pxx/LvR9Y4pz7TatVGTkmZlZsZkW+nQdMJziPMh8412+253g0j9O5wDy/JzUHuMBfWVMCjAPeSc636F7OueuccyOcc6MJ/n6Y55z7Fhk8Jvst7DPYX/SH4AqR5QTHRn8adj09/F3/DKwHGgmOWV5KcDzzJWAF8CIwwG9rwL1+XD4CSlt9zncIToCVA5eE/b26MB5fJTgE9CHwvv85PVPHBPgS8J4fj0XAz33/QQR/cZUDfwFyfH+uXy736w9q9Vk/9eO0DDgt7O/WTeNzAruvJtKYfM6PpqMQEZGUO0wkIiI9QGEgIiIKAxERURiIiAgKAxERQWEgIiIoDEREBPj/sj2yP6B45N4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "performance.net_worth.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this feedback loop is a bit slow for you, you can pass a callback function to `run`, which will be called at the end of each episode. The callback function will pass in a `data_frame` containing the agent's performance that episode, and expects a `bool` in return. If `True`, the agent will continue training, otherwise, the agent will stop and return its overall performance.\n",
    "\n",
    "## Saving and Restoring\n",
    "\n",
    "All trading strategies are capable of saving their agent to a file, for later restoring. The environment is not saved, as it does not have state that we care about preserving. To save our `TensorflowTradingStrategy` to a file, we just need to provide the path of the file to our strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "strategy.save_agent(path=\"agents/ppo_btc_1h\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This specific strategy saves multiple files, including a directory of models to the path provided._\n",
    "\n",
    "To restore the agent from the file, we first need to instantiate our strategy, before calling restore_agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_strategy = StableBaselinesTradingStrategy(environment=environment,\n",
    "                                          model=model,\n",
    "                                          policy=policy,\n",
    "                                          model_kwargs=params)\n",
    "\n",
    "new_strategy.restore_agent(path=\"agents/ppo_btc_1h\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our strategy is now restored back to its previous state, and ready to be used again. Let's see how it does.\n",
    "\n",
    "## Tuning Your Strategy\n",
    "\n",
    "Sometimes a trading strategy will require tuning a set of hyper-parameters, or features, on an environment to achieve maximum performance. In this case, each `TradingStrategy` provides an optionally implementable tune method.\n",
    "\n",
    "Tuning a model is similar to training a model, however in addition to adjusting and saving the weights and biases of the best performing model, the strategy also adjusts and persists the hyper-parameters that produced that model. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "from tensortrade.environments import TradingEnvironment\n",
    "from tensortrade.exchanges.simulated import StochasticExchange\n",
    "\n",
    "exchange = StochasticExchange(timeframe='1h',\n",
    "                       base_instrument='BTC',\n",
    "                       feature_pipeline=feature_pipeline)\n",
    "\n",
    "environment = TradingEnvironment(exchange=exchange,\n",
    "                                 action_scheme=action_scheme,\n",
    "                                 reward_scheme=reward_scheme)\n",
    "\n",
    "new_strategy.environment = environment\n",
    "\n",
    "tuned_performance = new_strategy.tune(episodes=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the agent will be trained for 10 episodes, with a different set of hyper-parameters each episode. The best set will be saved within the strategy, and used any time strategy.run() is called thereafter.\n",
    "\n",
    "## Strategy Evaluation\n",
    "\n",
    "Now that we've tuned and trained our agent, it's time to see how well it performs. To evaluate our strategy's performance on unseen data, we will need to run it on a new environment backed by such data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished running strategy.\n",
      "Total episodes: 1 (2000 timesteps).\n",
      "Average reward: -0.7945276696372455.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from tensortrade.environments import TradingEnvironment\n",
    "from tensortrade.exchanges.simulated import SimulatedExchange\n",
    "\n",
    "df = pd.read_csv('data/Coinbase_BTCUSD_d.csv', skiprows=1)\n",
    "exchange = SimulatedExchange(data_frame=df,\n",
    "                             feature_pipeline=feature_pipeline,\n",
    "                             base_instrument='USD',\n",
    "                             pretransform=True)\n",
    "\n",
    "environment = TradingEnvironment(exchange=exchange,\n",
    "                                 action_scheme=action_scheme,\n",
    "                                 reward_scheme=reward_scheme)\n",
    "\n",
    "new_strategy.environment = environment\n",
    "\n",
    "test_performance = new_strategy.run(steps=2000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1b2fbf350>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD4CAYAAAAdIcpQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hc1bXw/++aGY26NOpWsyTbko0r7qaYjjHVtBBKaCEhCSSEJO9N4Ob+ftyQ8oYkJDfkAoEEEwhgh9BbMMY4dNx7lVxVbBWr9zL7/WOOhGSrWJqRZsZen+fRo5l9yqw59szSLmdvMcaglFJK9cfm7wCUUkoFPk0WSimlBqTJQiml1IA0WSillBqQJgullFIDcvg7gKFKTEw02dnZ/g5DKaWCRmJiIsuWLVtmjFk42GODNllkZ2ezdu1af4ehlFJBRUQSh3KcNkMppZQakCYLpZRSA9JkoZRSakCaLJRSSg1Ik4VSSqkBabJQSik1IE0WSimlBqTJQo2Yz/cc4bM9Ff4OQyk1BEF7U54KPt9fuoGyuhYunZLKA5dPJDkmzN8hKaWOk9Ys1Igoq2umrK6F2dlxvL+jlB+/vNnfISmlBkGThRoR20pqAfjRgvFceWo6m4tq/ByRUmowNFmoEbHdShYT02LITYmisqGVivoWP0ellDpemizUsCkoq6elvQOAbSU1jI6PICYshLyUaADyS+v9GZ5SahA0Wahh8dHuchb84UP+7zs7AU8z1KS0GIAvk0VZnd/iU0oNjiYL5XOFlY3cs3QDbgOvbSymsqGVA0cau5JFSkwo0WEOdpdqslAqWAyYLERksYiUicjWbmW/FZGdIrJZRF4VEVe3bfeLSIGI7BKRi7qVL7TKCkTkvm7lOSKyyir/h4g4ffkG1ciqa27jW39fh9tteODyiVQ3tvH4vwsAmJQWC4CIkJscxW5thlIqaBxPzeJvwNGrKi0HJhtjpgK7gfsBRGQicD0wyTrmMRGxi4gdeBS4GJgI3GDtC/AQ8AdjzDigCrjDq3ek/KahpZ3bn17DrtI6/njDdG6el0VilJNnPjsA0FWzAE9TVEGZJgulgsWAycIY8xFQeVTZe8aYduvpF0CG9XgRsNQY02KM2QcUAHOsnwJjzF5jTCuwFFgkIgKcB7xkHf8McKWX70n5QWu7mzueWcOGwmoeuX46545PxmG3cfm0NFo73CRGhfa4CS83JVpHRCkVRHzRZ/F14F/W43SgsNu2Iqusr/IEoLpb4uksV0HmjU0lfLG3kl9fPYVLp6Z2lV893fN3xMRutQqAvJQoAO23UCpIeJUsROSnQDvwvG/CGfD17hSRtSKytry8fCRe8qRVUt3ExsLq49rXGMNfPtrLhFHRXDszo8e2yekxXDEtjStPTetRrsNnlQouQ04WInIbcBlwkzHGWMXFQGa33TKssr7KjwAuEXEcVd4rY8yTxphZxphZSUlJQw1dDcAYwzefXcu1j3/G+oNVA+7/UX4Fu0rr+Mb8MXhaFr8kIjxyw3SuntEziSRHhxIT5tDhs0oFiSElCxFZCPwYuMIY09ht0xvA9SISKiI5QC6wGlgD5Fojn5x4OsHfsJLMSuBa6/hbgdeH9laUr6zYUca2klocduG7z6/v6ltYva+SrcU17CmvZ1tJDZsKqymrbeYvH+0lJSaUK6alDXxyi4iQmxKtI6KUChIDzjorIkuAc4BEESkCHsAz+ikUWG79JfmFMebbxphtIvIisB1P89TdxpgO6zzfBZYBdmCxMWab9RI/AZaKyC+ADcBTPnx/agCFlY04HTZSrM5nYwyPfJBPZnw4j1w/na8++QXnPfxvqhvb+j3PjxeOx+kY3N8e45KiWLGzbMixK6VGzoDJwhhzQy/FfX6hG2N+Cfyyl/J3gHd6Kd+LZ7SUGmFut+HaP39GVWMbN8/L4tKpqew8VMfmohoeumYK00fH8ZtrpvLKhmLm5sQzKS2G5jY3zW0dhIXYsduEwzVNVDe2cdvp2YN+/azECCrqW6hvaScqVGfLVyqQ6Sf0JLbjcC2ltS1My4jl6U/38dQn+wDIiAvv6mO4cno6V04fngFqOQmRAOyvaGByeuywvIZSyjc0WZzEPi3wrFr3xM2zaG7rYN+RBsAzUinEPvwzwWRZyeLAkUZNFkoFOE0WfmSMobqxjSMNraS7wgl32imubuKfawuZn5vEzKy4fo9vbXezr6KBhCgn8RFObDbpd/+jfZxfQW5yFKNiPf0V2YmRQ34vQ5GdGAHAfitJKaUClyaLEdTW4eZQdTO7S+t4e8shlm8vpb7Fcz+iwyaMTYqioLyeDrfh8z1H+Me3Tuv3fA+/t4snPtoLQGpsGC9953TSXeHHFUtzWwer91Vy49zR3r0pL0Q4HSRHh7K/QpOFUoFOk8UI+Ti/nDufXUdTm2d9h9jwEC6dkkreqGjiI0MoKKtnc1ENZ49Poq65nX+sOciR+hYSokJ7PV9zWwdL1xRy+tgELjglhd+9t4ufvLSZv98x55h7HXqz7kAVLe1u5ucm+vR9DlZ2YqTWLJQKAposhsnBI428ubmEO87Moa3DzY9f2kyqK4xvnzWWzPgIZmbF9TnUdGtxDUtWH2TFzjKum5XZ6z5vbT5ETVMb3z1vHKePTSQ0xMZPX93Kc6sOcvO8rAHj+zi/ghC7MDcnwav36a3shAhW7tK78ZUKdJoshslDy3by9uZDrNhRSlZCJKW1zbz8ndOZPrr/fgjwzM6a7grnvW2H+0wWz31xgLFJkZw2xvNlf+Oc0by79TC/ensHeclRzB3TfxL4pKCc6aPjiPTzkNXsxEjK1xbp8FmlApwufjQMjtS38N62w8zJjmdbSS2vbijm62fkHFeiAM/dzQsmpfBRfgUNLe3HbN9aXMPGwmpumpvV1eQkIjz8lWmkucK4+anVvLmppM/zV9S3sLW4lvnj/NsEBZDdbfisUipwabIYBq9uKKatw/CLqyaz9M553HZ6Nj9aMH5Q51gwcRSt7W4+2t2zicYYw/+8n09YiI1rjpq0LzkmjJe/czrTMmP53pINfPvv69h5uPaYc3ee85zxyYN8Z76X3W34rFIqcGmy8DFjDEtWH2TGaBd5KdFMHx3Hf18xiXCnfVDnmZ0dR3ykk0f/XUBZbXNX+d8+28/7O0r5PwvGExsecsxxrggnf79jLvecn8unBRVc/MePeWRFPl/O9Qgrd5WTGOXssRiRv2Ql6PBZpYKBJgsfW3ugij3lDVw/27shqQ67jV9dNYU9ZQ1c9qdPeO6LAzz3xQF+9c4Ozp+QzB1n5vR5bFiInR9emMfHPzmXq05N5/fLd3PP0o00t3XQ4TZ8nF/O2XnJg74vYzhEhurwWaWCgfYo+tj7O0oJsUuPBYCGauHkUeQknsG3n1vHf73mWQI93RXO774y7biGx7oinDx83TRyU6J56N2dRIc5uGZGBtWNbZwzPnCmeM9O0OGzSgU6TRY+VtvUhivC6bNRRuNHRfPeD86itLaZ1nY3o2LDiHAe/7lFhO+cM5bqxlae+GgvBaX12AS/31/RXVZCBP/ercNnlQpkmix8rL6lw+dDQEPsNjLiIrw6xw8X5PHh7nJW769kZlYcrginj6LzXpornIr6Flrb3YOe5lwpNTL0k+lj9c1tAXm/QKjDzh++eipOh42LJqX4O5we0lxhGANldc0D76yU8ovA+1YLcoF8c9kpqTF8dt95uHoZReVPo2I981kdrmn2ugallBoeWrPwsfqWDr/fFd2fxKhQHCMw/fhgpFqz3h6qCbyaRVNrB2f8+gP+vUtX9FMnt8D61jgB1Le0ER0WuMkiEI3qShZNfo7kWOV1LRRXN/H5niP+DkUpv9Jk4WP1zYHbDBWookMdRDrtAVmz6JxCfq/eB6JOcposfKwhwJuhApGIkOoK53AAJouGVitZlNf7ORKl/GvAZCEii0WkTES2div7iohsExG3iMw6av/7RaRARHaJyEXdyhdaZQUicl+38hwRWWWV/0NEAmdM5yC1tHfQ2uHWZqghSI0NC+iaxcHKRto73H6ORin/OZ6axd+AhUeVbQWuBj7qXigiE4HrgUnWMY+JiF1E7MCjwMXAROAGa1+Ah4A/GGPGAVXAHUN7K/7X0OJZ2ChykPNAKRgVExaQfRads/62dRiKqgIvPqVGyoDJwhjzEVB5VNkOY8yuXnZfBCw1xrQYY/YBBcAc66fAGLPXGNMKLAUWiWfOivOAl6zjnwGuHPK78bP6Zs8XS1RYYA1NDQapsWGU1bXQFmB/vXefIn6f9luok5iv+yzSgcJuz4ussr7KE4BqY0z7UeVBqa6lDUA7uIdgVGw4xnhGHwWSequ2CLCnW79FS3sHj6zIZ/n2Un+EpdSIC6pvNRG5E7gTYPRo72Z1HQ6dzVCaLAYv1fXlvRZprnA/R/OlzppFdKijq2axr6KB7y1Zz9biWqJDHaz40dkkx4T5M0ylhp2vaxbFQPd1QDOssr7KjwAuEXEcVd4rY8yTxphZxphZSUmBM2tqp/rOmoV2cA9a5415gTYiqqGlnVCHjXEpUewtb6C5rYPrnvicwsomfnbFJFra3fzqnR3+DlOpYefrb7U3gBdE5PdAGpALrAYEyBWRHDzJ4HrgRmOMEZGVwLV4+jFuBV73cUwjpl5rFkOWGuOpTQRaJ3fn9C05iZF8VnCEtzYforyuhb/fMYf5uUkcqW/hkQ8KOHdCMqdmuqhrbmd7SS3JMaEBsRKhUr4y4LeaiCwBzgESRaQIeABPh/efgCTgbRHZaIy5yBizTUReBLYD7cDdxpgO6zzfBZYBdmCxMWab9RI/AZaKyC+ADcBTvnyDI6mrg1uTxaDFhDsIDwm8G/MaWtqJDHUwNimKV9YX8+RHexibFMmZ1vrld507jtc2lvD9pRt7HBfptLP5vy/CHgALTCnlCwN+qxljbuhj06t97P9L4Je9lL8DvNNL+V48o6WCnjZDDZ2IkBob5vNmqJb2DnYfrqe2uY2MuHCyrDW/j1fnXF85iZ7jdpfW8+CiSV2LT4WF2Hn97jNYf7CKqsY2Qh02DlY28ttluygoq2f8qGifvh+l/EW/1XyovqUDEYgI0fsshiLV5dt7LYwx3PLUalbt84z8To4O5bP7zhvURIoNLe1EhdoZk+RJFlGhDq6ekdFjn7hIJ+ef8uW07wVl9fx22S42FVZrslAnDJ3uw4fqm9uJdDoCYm3rYDQqJtynzVDv7yhj1b5K7j53LPdfPIGyuhY+KagY1DnqrWao7IRIwkJsfGVWxoDNjGMSI4kOc7CxqNqb8JUKKJosfKi+JTAXPgoWY5MjOVTTzCf5g/tC702H2/C7ZbvISYzk3gvyuO2MbGLCHLy2oc/Bdr3q7LMIC7Hz9j3z+cnCCQMeY7MJ0zJcbDyoyUKdODRZ+FBDS4f2V3jhttOzyUuJ4vtLN3jdd/HmphJ2ldbxgwvzCLHbCHXYuXRqGsu2lfa4K3sg9S3tRFlrno9NiiLsOJsYT810sau0jqbWjoF3VioIaLLwoTrrr1A1NBFOB4/dNJPmtg6+t2Q9xpghn2vxp/uYMCqay6akdpVdPSOdprYO3tt++LjP0zDEf9NpmS463IatJTWDPlapQKTJwocaWtqJ1mThlXHJUfxowXjW7K8a8lxMHW7DzsN1zM9N7NF/NHN0HBlx4byy/viaotxuQ0NrB1Ghgx+wMC0zFoBNhdoUpU4Mmix8qL65ncghfLGonuaOiQdgW0ntkI4/cKSB1nY3uSk9RyLZbMINc0bzcX4FL60rGvA8jW3WLMJD+AMgOTqMdFc4GzRZqBOEJgsf8tztqzPOeis3ORqn3TbkJpzdpZ4J//JSjh22+q2zxnDamAT+89UtbCnq//ydfRtDbVqclhmrndzqhKHJwofqW9p14SMfcDps5I2KYvsQaxYFZXUA5CZHHbPNYbfxvzdOJykqlLteWIfb3Xe/SOfCR0Md4TYzK57i6iaKqhqHdLxSgUSThY8YY6wx+doM5QuT02LZWlwzpE7u3aX1pLvC+6wRJESFcvsZ2RRWNlHX3PfIKG9rFmfleqYE8cVQYKX8TZOFjzS3uelwG22G8pFJaTFUNbZRMoQhtLtL68hLObZW0V1ClGf13srG1j738bZmMS45ipSYUD7WZKFOAJosfKTri0WboXxiUrpnNNG24sH1W7R3uNlb3tBrf0V3cRFWsmjoO1l4uz6JiHDmuCQ+3VNBRz/NXUoFA00WPvLlX6HaDOULp4yKwSawdZD9FgcqG2ntOHYk1NHiIz3JouqoZFHT1MYP/7GR6sbWbs1QQ/83PSsvkerGNrYOMukpFWg0WfhIQ1ey0GYoXwh32hmbFMX2XkZE7Smv55zfruQ//rmJA0d63ouRX+rp3B6oGaqzZlF1VDPU+oNVvLKhmM/3HPG6GQrgDGsq8845qeqa27y62VApf9E2Ex+p07UsfG5yeiyf7znSo6yxtZ3vPLeOI/WtvL6phFc2FPP766ax6FTP0u2dw2bH9TISqru4yN6TRU2jZ5r5A5WNdN7O581d+YlRoUxMjeGNjSV8nF/OF3sriQp1kJcSxV9umUVCVOiQz63USNJvNh/xxV+hqqdJaTG8uqGYy//0CcnRoUxKi2Hn4Tryy+p59utzGJ8SzZ1/X8fP3tzOOXnJxEaEsLu0jsz4cCKc/f87RDrtOO02KhvaepTXNHmeH6xsJDHS6Zly3uld0+L8vESe+HAvSdGh3HPeOA7XNvPi2iI+KajoSnJKBTr9ZvORBu3g9rnLp6WRX1pPaV0zRVVNrNxVhtvADy/MY36uZw32X141mcv/9Al/eH83d541hvUHqjglNWbAc4sIcZEhx/RZVFs1i8LKRsIcdiKdjq6FjobqO2ePZWJqDBdNGkVYiJ32DjdvbT7E+gNVmixU0DghvtkWf7KPnMRIzp3gvzWP67Rm4XMpMWE8dO3UrucNLe0crGxkQrcFhSalxXLT3Cye/Xw/L68rot1t+Nq8rOM6f1yE85ihs91rFqmxYT65b8YV4eyRFBx2G9MyXKw7WOX1uZUaKUHfwX3gSAMPvrWdb/19Hev9+OHT9beHX2Sog1NSY475S/9HC/JIjg5jYloM7947/7j/aIiPdB5bs2jyPC+uaqKmqW3YZhGekeVix6E6GluPf7p0pfwp6JPFC6sPYrcJSdGh3PnsOp8uyzkYDS3t2G1CWEjQX9Kg44pw8slPzuUf3zptUGtsx0U6j+ngrrVqFu1uQ35Z/bAl/5lZcXS4DZsHmJ9KqUAx4DebiCwWkTIR2dqtLF5ElotIvvU7zioXEXlERApEZLOIzOh2zK3W/vkicmu38pkissU65hEZRANxS3sHL60t4vwJyTx9+2ya2zq46a+rKKwc+bl46lvaiXTavW7fVkMzmHW1O8VFhFDVeGwHt9M61/6KBiIH6CgfqumZcQCsO6BNUSo4HM8n7G/AwqPK7gNWGGNygRXWc4CLgVzr507gcfAkF+ABYC4wB3igM8FY+3yz23FHv1aflm0r5UhDKzfNyyIvJZrFt82moq6Fqx77jI/zy2nrcB/vqbzS4TasP1hFog6DDCrxEU6qG1t73F1d3djGhFRPn4jbeDdstj9xkU7GJEayQfstVJAYMFkYYz4CKo8qXgQ8Yz1+BriyW/mzxuMLwCUiqcBFwHJjTKUxpgpYDiy0tsUYY74wnjuVnu12rn7VNrfx53/vITM+nPnWjU9zcuJ55a7TCXXYuPmp1Ux/cDk3/fULfvrqFt7devyrow3WXz7ey+aiGr5/Qe6wvYbyvbhIJ27zZdMTeGoW41OiCbF7aojDeUf+jKw41h+s1pv0VFAYagN7ijHmkPX4MJBiPU4HCrvtV2SV9Vde1Et5r0TkThFZKyJrDxxpZFdpHfecl9tjNbRxydG8e+98HrtpBotOTaO+uZ3XN5Zwz5INNLf5fj3k/NI6fr98NwsnjeKKaWk+P78aPp1TfnQfEVXd1EZ8pJPMuAhg+GoWADNGx1HZ0Mq7Ww+PWC1YqaHy+pNgjDEiMiJ/GhljngSeBJg49VSz7r8vIryXG6aiw0K4ZEoql1jrL7+/vZRvPLuWjYXVzBuT4NOY/vD+biKcdn5+5WTtrwgyXVN+NLRCEjS3ddDa7iY2IoTM+Aj2VjQM6+i2s8cn4YoI4TvPryc2PIRfXz2Fi7utGa5UIBlqzaLUakLC+l1mlRcDmd32y7DK+ivP6KV8QBFOR6+Jojezs+MRgdX7jm5N8962klrOGJdIUrT2VwSbrskErU7uzhvyYsNDyEoY/ppFuiucz+47jydunklOYiTfXbKBNzeVDNvrKeWNoSaLN4DOEU23Aq93K7/FGhU1D6ixmquWAQtEJM7q2F4ALLO21YrIPGsU1C3dzuUzsREhjE+J9nmyaG7r4GBlY68rsqnA54rwTPrYea9F5w15rnAno+OHP1mA54+eiyaN4vlvzGVmVhzfX7qBTwt0/QsVeI5n6OwS4HNgvIgUicgdwK+BC0UkH7jAeg7wDrAXKAD+AtwFYIypBH4OrLF+HrTKsPb5q3XMHuBfvnlrPc0bk8C6A1U+bRveW96AMQNPWqcC09F9FtXW79hwTzMUjNyU85GhDv52+2yiQh1au1ABacA/m4wxN/Sx6fxe9jXA3X2cZzGwuJfytcDkgeLw1pyceP722X62FtcwfXRcV7kxpkdfQ01jG4VVjdQ2tTErOx6no+98mt+11nP/ayeowBQeYifUYTumZhEbHkJGXDginilHRkqE08Hs7HhW7/d9c6lS3jpp5qaYnR0PePotOpPFBztLuev59Vw7M4OvzMzk+VUHeGldEZ3D7i+alMLjN83sMdqqu4Kyeuw2ITsxYkTeg/ItESE+0tm1Wl5XM5TVwb3yR+d09V2MlNk58azYWUZFfYvet6MCykkzN0VSdChjkiK7+i3cbsND/9pFWIidpasLWfTop7y2oYRbTsvmz1+byb0X5LJsWykPL9/V5zkLyurJio8g1KGr4wWruIgvp/zoTBYx4Z6+jOzEyBEf4db5R81arV2oAHPS1CwA5ubE8+amQxRVNbK5qIZdpXX88fpTmTE6jvd3lHLhxBQyrPH1F01K4XBNM4+u3INdhLvOHUdYSM+kkF9Wr/0VQS4+0tk1GqqmqQ2bQLQfJ4Ockh5LWIiN1fuqWDhZh9GqwHFSJYtbTsvm7c2H+OoTXxDqsDEmKZLLpqZhtwm3n5HTY18R4cFFk2ls7eCRDwp4eX0xeSlRFFU18ZVZGdx+Rg77Kxq4aFJKH6+mgoErIoSSEs/kk9WNbcSEh/TZ7DgSnA4bp2a6WKM1CxVgTppmKIBTUmN44ZvzaGxtZ29FA98/Pxd7P18MToeNR26YzgvfnMuo2DBKa1todxv++H4+m4uqaXcbrVkEufjIL9e0qGlqwxXu/zXU52THs62kpmv1RaUCwUlVswDPus7//PbprNxZxmVTj296jtPHJnL6dzzzT20vqeWSRz7mwTe3AzoSKtjFRTipaWqjvcNNdVMbsQGQLGbnxOP+ANYfqOKsvCR/h6MUcJLVLDqNS47im2eN6bdW0ZeJaTGclZfEpqIaRGBsktYsgllClBNjoKK+lZqmNmKtKUD8acboOOw20U5uFVBOymThrW+fPQbwTNdwvFOOqMA0NcMFwJr9ldQGSM0iMtTB2KRIth+q9XcoSnU56ZqhfOG0MQnMyY4n1TVyN2yp4TElPZboMAefFlRQ3dhKbHhgfCQmpsYMy1xmSg1VYHwygoyI8Nw35g6pGUsFFrtNOG1MAp8UVFDb3I4r3P/NUOAZjPHaxhKqG1txBUDTmFLaDDVETodNk8UJ4szcRIqqmuhwm4BohgJP3xigTVEqYGiyUCe9M6yVFsEzQ3EgOCXVkyx2HKrzcyRKeWiyUCe9MYmRpMZ6+p8CpWaRGBVKcnQo20u0ZqECgyYLddITEU4f66ldBMJNeZ1OSY3RZigVMDRZKAWcNyEZgNTYcD9H8qVTUmMoKKujtV3X51b+p8lCKeCSKaNY8aOzGT3CU5L3Z2JaDG0dhoKyen+HopQmC6XA0xQVaHfjT0z1TCWzQ5uiVADQZKFUgMpJjCIsxMaW4hp/h6KUJgulApXdJszNSWDFzlI8KxYr5T+aLJQKYJdOSaWwsomtxdoUpfzLq2QhIt8Xka0isk1E7rXK4kVkuYjkW7/jrHIRkUdEpEBENovIjG7nudXaP19EbvXuLSl14lgwKQWHTXh7yyF/h6JOckNOFiIyGfgmMAeYBlwmIuOA+4AVxphcYIX1HOBiINf6uRN43DpPPPAAMNc61wOdCUapk50rwsnp4xJ5Z8uhHk1RhZWNbNW+DDWCvKlZnAKsMsY0GmPagQ+Bq4FFwDPWPs8AV1qPFwHPGo8vAJeIpAIXAcuNMZXGmCpgObDQi7iUOqFcOmUUBysb2dbtbu77XtnMN59d68eo1MnGm2SxFZgvIgkiEgFcAmQCKcaYzjrzYaBzkep0oLDb8UVWWV/lxxCRO0VkrYisLS8v9yJ0pYLHgomjsNuEtzZ7Pla1zW2s2lvJoZpmDtc0+zk6dbIYcrIwxuwAHgLeA94FNgIdR+1jAJ8N4zDGPGmMmWWMmZWUpMtNqpNDXKSTs3ITeW1DMe0dbj7eXUG72/Ox2lRU7efo1MnCqw5uY8xTxpiZxpizgCpgN1BqNS9h/S6zdi/GU/PolGGV9VWulLJ8dfZoDtc28+9d5azYUUpseAgOm7BZk4UaId6Ohkq2fo/G01/xAvAG0Dmi6VbgdevxG8At1qioeUCN1Vy1DFggInFWx/YCq0wpZTn/lGSSokN5btUBVu4q4/wJyeSlRLO5SDu51cjwdqW8l0UkAWgD7jbGVIvIr4EXReQO4ABwnbXvO3j6NQqARuB2AGNMpYj8HFhj7fegMUbXk1SqmxC7jetmZfDoyj0AnHdKMqEhdt7eXIIxBhFdiEsNL6+ShTFmfi9lR4Dzeyk3wN19nGcxsNibWJQ60V0/ezSPrtyDwybMz02ivrmdJasPsv9IIzmJkf4OT53gdA1upYJEZnwEF08e1bX869QMFwCbCqs1Wahhp8lCqSDy2E1dEx+Ql+KZaHBTUe9+DyMAABnsSURBVDVXTu91tLlSPqPJQqkg0r1vwmG3MTktlk2FOiJKDT+dSFCpIDZ3TDwbC6vZVqKjotTw0mShVBC7c/5Y4iKc/NdrW3G7dRpzNXw0WSgVxGIjQvjPS05hw8FqXlxbOPABSg2RJgulgtzVM9KZkx3Pr9/dSWVDq7/DUScoTRZKBTkR4RdXTaa+uZ1f/2uHv8NRJyhNFkqdAPJSorljfg4vri1i7X6dAEH5niYLpU4Q95yXS1psGP/12lY6tLNb+ZgmC6VOEJGhDu45P5edh+soKKv3dzjqBKPJQqkTyKxsz4rEW3TJVeVjmiyUOoHkJEYR4bT7ZH3uNfsrWbGj1AdRqROBJgulTiB2mzApLcYnNYvfvLuT7y3ZQHWjDsdVmiyUOuFMTo9lW0kN7R1ur85TUFZPY2sHz31xwEeRqWCmyUKpE8yU9Fia29zsKW8Y8jkqG1qpamzDYROe/nQ/zW0dPoxQBSNNFkqdYKakxwLedXJ3jqb65lljONLQyj/XFfkkNhW8NFkodYIZk+R9J3dnsrhxzmimZbr42RvbuPxPn/DUJ/v6POZQTRNvbS6hvqV9yK+rApeuZ6HUCcZuEyametfJvae8nvAQO+mucB69cTovrDrIh7vL+flb21kwMYXM+AhW7irjnhc2MDohgpiwEFbtO4LbwJikSP78tZnkpUT78F0pf9OahVInoMnpsWwvqR3yndwFZfWMSYrEZhMy4iL48cIJ/PlrMwF4e8shAJ7+dD9Oh42EqFCqGlu565xxPHrjDGqb2rnifz/he0s28MKqg153tKvA4FWyEJEfiMg2EdkqIktEJExEckRklYgUiMg/RMRp7RtqPS+wtmd3O8/9VvkuEbnIu7eklJo+2kVTWwcPv7drSOtc7CmvZ2xSVI+yzPgIpmW6eGtzCYdrmvkkv5yb5o7m2a/P4d17z+L/XDSeS6em8s49Z3LplDRW7zvCf766pd+mKxU8hpwsRCQduAeYZYyZDNiB64GHgD8YY8YBVcAd1iF3AFVW+R+s/RCRidZxk4CFwGMiYh9qXEopuGRKKtfNyuCxf+/he0s3DGo0U1NrB8XVTYxLjjpm22VTUtlaXMsflu/GbeCamRnH7JMcE8bD103ji/vPZ8KoaD7Or+ix3RjDZ3sqOHDk2NFau0vreHfr4eOOVY0cb5uhHEC4iDiACOAQcB7wkrX9GeBK6/Ei6znW9vPFs6DwImCpMabFGLMPKADmeBmXUie1ELuNh66Zyv0XT+CdLYf4xjNraWw9vo7nvRX1GMMxNQuAS6amAvCPtYXMyY4nKyGyz/OICPPGJLD2QCUt7Z5ktetwHTf9dRU3/mUVv3qn53TqNU1t3Lp4Nd95fh27S+uO962qETLkZGGMKQZ+BxzEkyRqgHVAtTGm839lEZBuPU4HCq1j2639E7qX93JMDyJyp4isFZG15eXlQw1dqZOCiPCts8fy22un8dmeCm5bvIbV+yoHrGV0joTqrWaR7gpnxmgXANf2Uqs42mljE2huc7OpsIbmtg6uf/Jzth+qJS02jEM1zT32ffDN7ZTVtRAeYufh93Yd79tUI8SbZqg4PLWCHCANiMTTjDRsjDFPGmNmGWNmJSUlDedLKXXCuHZmBn+8fjobC6u57onPmfaz93jiwz0Y03tfxp6yemwCWQkRvW6/cW4WabFhXbWM/szLSUAEPt9zhDc3lVDV2MbjN83kzNxESmu/TBbvby/l5fVF3HXOWL511liWbStlU2H10N6wGhbeDJ29ANhnjCkHEJFXgDMAl4g4rNpDBlBs7V8MZAJFVrNVLHCkW3mn7scopXzg8mlpnDkukTX7K/nnuiL+7792sru0nl9dPZlQR88uwj3lDWTGRxAW0nvX4bUzM46rVgGeNcInpsbw+d4KmtvcjE2KZN6YeD7bU0FFfSsdboPdJixZfZCMuHC+d14urR1unvl8P/e9soVzxicxOj6CG+aM9vYSKC9502dxEJgnIhFW38P5wHZgJXCttc+twOvW4zes51jbPzCeP23eAK63RkvlALnAai/iUkr1Ii7SyYJJo3jy5pnce0EuL68v4oo/fcrmoi//gne7DesPVnHKqBifve5pYxJYva+SjYXV3DQ3CxEhOTqUDrfhSEMLAMXVTUwYFYPTYSMq1MH/d9kplFQ38ZeP9nL/K1vYXzH0qUuUb3jTZ7EKT0f1emCLda4ngZ8APxSRAjx9Ek9ZhzwFJFjlPwTus86zDXgRT6J5F7jbGKMT0Sg1TESEey/IY/Fts6huauWqxz7j1Q2e6TzW7K/kUE0zF08Z5bPXO21sAm4DYSE2rpnhqZEkx4QBUFb7ZbJId4V1HXPV9Aw2PbCAN757JgCbirRJyt+8Gg1ljHnAGDPBGDPZGHOzNaJprzFmjjFmnDHmK8aYFmvfZuv5OGv73m7n+aUxZqwxZrwx5l/eviml1MDOm5DCez84mynpsTz0r120tHfwxqYSwkPsXDgxxWevMzsnnhC7cPnUNGIjQgBIjg4FoKyumdrmNuqa20lzhR9zbF5KFGEhNjZq/4Xf6R3cSp3EYsND+NGCPA7XNvPi2iLe2XKICyamEOH03UxAMWEhvPTt0/mvyyZ2laVYNYvS2hYOVXs6untLFg67jSnpsdrZHQA0WSh1kjtzXCJT0mP55dvbqWps44ppaT5/jWmZLmLDQ7qeJ1k1i9LaZkqqm4DekwXAtAwXW0tqadNpQ/xKk4VSJzkR4e5zx9Lc5iYmzMFZeYnD/pohdhsJkU7K6lootpJFel/JItNFa7ubXYf1Rj1/0llnlVIsmDiKaZku5mTHHTOUdrgkx4RRVtuMKzwEh026ahtHOzXTcxPgxsJqJltrdaiRp8lCKYXNJrx+9xkj+popMaGU1rYQFepgVGwYdpv0ul9GXDgJkU42FlbztXlZIxqj+pI2Qyml/CI5OtTqs2jus78CPM1k0zJdXZ3cxhg+zi/n/lc2s+NQ7UiFe9LTmoVSyi9SYsKoqG/BYRPmjknod99pGS5W7irj9qdXU1jV1DV/1WsbSnjo2qnD0imvetKahVLKL5JjwnAbKKlpJq3bDXm9WTh5FFPSY6mobyUpKpTfXDuVT35yLpPSYrhnyQY+2Fk6QlGfvLRmoZTyi+RuHdr9NUMBjB8V3XU3d3cvfHMeM36+nA92lnHeBN/dSKiOpTULpZRfdN6YBwMni744HTamj3axdn+Vr8JSfdBkoZTyi+41i77usTges7Li2VVaR01Tmy/CUn3QZKGU8ovu91WkxvbfZ9Gf2dlxGAPrD2rtYjhpslBK+UWI3UZilJOYMAfRYSEDH9CHU0e7sNuEdcfZFPWbd3cy4+fLefi9XZTXtQz5dU82miyUUn6TFB025P6KThFOB5PSYlizv/K49v9wdzlt7W7+d2UBlzzyMfUtx7c2+clOk4VSym9unJPJjXO9XwVvVlY8GwuraW3vf7LB1nY3u0vruGleFku/OY/yuhae+Wy/169/MtBkoZTym5tPy+aW07K9Ps/s7Dha2t1sK6npd7/8sjraOgyT0mKYOyaB8yck8+RHe6lt1s7xgWiyUEoFvZnZcQA8urKAhn6albaVeKYHmZTmWTb2BxfmUdPUxtOf7B/2GIOdJgulVNBLjg7jvy49hQ92lnHN4591TXt+tO0ltUQ47WQnRAIwOT2WBRNT+OOK3Zz/8L/54T82cqReO717o8lCKXVC+Mb8MTx9+xwKKxv5xVvbe91nW0kNp6TGYOs2w+2vrp7CXeeMY1xyFG9vOcRVj33WNffUUB2qaeLNTSV8vucIFSdI8tHpPpRSJ4yz85L42rws/vrJPg7XNDOq2/0bbrdhe0kt18zM6HFMYlQo/+ei8YBnzYxvPLOGqx/7lD/fPJPTxw5+IaiaxjaufuwzDtV4losNC7Fx38IJ3HJado8kFWyGXLMQkfEisrHbT62I3Csi8SKyXETyrd9x1v4iIo+ISIGIbBaRGd3Odau1f76I3OqLN6aUOjndOHc0bmNYsvpgj/IDlY00tHZ09Vf05tRMF6/edQYpMWHc8tRq/rm28Lhe86F3d/LLt7fT0NLOT1/bQnldC3+5ZRbP3TGXeWMS+O83tzP/NyuZ/uB7nPnQB+wt967m4g9DThbGmF3GmFONMacCM4FG4FXgPmCFMSYXWGE9B7gYyLV+7gQeBxCReOABYC4wB3igM8EopdRgZSVEcnZeEktWH+yxbvf2rs7t/lfby4yP4OW7Tue0sQn8x0ub+f3y3Rhj+ty/sLKRP3+4h798vI+zfrOStzYf4gcX5nHhxBTOzE3k6dtm85trpjIpLYaLp6TS2NrB1/+2hqqGVt+84RHiqz6L84E9xpgDwCLgGav8GeBK6/Ei4Fnj8QXgEpFU4CJguTGm0hhTBSwHFvooLqXUSejmeVmU1bXwxId72Fpcw76KBj7fW4HDJuSmRA14fExYCItvm811szJ4ZEU+//HSZppaO3rd95/rigB45IbpuCJCmJ+byLfPHtu1XUS4bnYmT94yi19dNYUnb55JSXUz335uHW5330ko0Piqz+J6YIn1OMUYc8h6fBjonDc4HehepyuyyvoqP4aI3ImnVsLo0d7fyKOUOjGdMz6ZCaOi+d17u/nde7u7yienxxz3GuMhdhsPXTOVNFc4//N+Psu3l/LV2ZlkxIXT1mG4ePIoUmLCeGltIWeOS+SKaWlcPjUVY+i3b2JWdjw/XjieX7y9g70V9YxLjvb6/Y4Er5OFiDiBK4D7j95mjDEi4rPUaYx5EngSYNasWcGTkpVSI8puE1696wwKyuoprm6kqa2DELuNKen9N0EdTUS494I85ucm8tQn+/jrx3vprAw8+dEevn32WEpqmvnPS0/p2l+Oow87L8WTICobgudmQF/ULC4G1htjOpeqKhWRVGPMIauZqcwqLwYyux2XYZUVA+ccVf5vH8SllDqJhTvtTMmIZUrG4BJEb2ZmxTMzK56apjbaOtyUVDfx9b+t4WdvbicuIoQLJw5u4aW4CCcAVY3B02/hiz6LG/iyCQrgDaBzRNOtwOvdym+xRkXNA2qs5qplwAIRibM6thdYZUopFVBiw0NIjAplaoaLpXeexuj4CG4/I+e4m7Y6uSI8s+zWNJ4kNQsRiQQuBL7VrfjXwIsicgdwALjOKn8HuAQowDNy6nYAY0yliPwcWGPt96Ax5vimj1RKKT8ZlxzFh/9xzpCOjYsMvpqFV8nCGNMAJBxVdgTP6Kij9zXA3X2cZzGw2JtYlFJqpMnxdFD0ItJpJ8QuVAVRzUKn+1BKqREmIrginFQHUc1Ck4VSSvlBXERIUDVDabJQSik/cEU4tRlKKaVU/+IiQrQZSimlVP/itGahlFJqIJ0d3P1NUhhINFkopZQfuCJCaOswNPQxQWGg0WShlFJ+EGfdxR0sU5VrslBKKT9wWfNDVQdJv4UmC6WU8oNgm0xQk4VSSvlBVzOUJgullFJ90WYopZRSA3JpzUIppdRAQuw2okMdWrNQSinVP1dk8Ez5oclCKaX8JJim/NBkoZRSfhJMa1poslBKKT/xrGmhNQullFL98DRDac1CKaVUP1wRIdQ1t9Pe4fZ3KAPyKlmIiEtEXhKRnSKyQ0ROE5F4EVkuIvnW7zhrXxGRR0SkQEQ2i8iMbue51do/X0Ru9fZNKaVUMOic8qO6KfCborytWfwReNcYMwGYBuwA7gNWGGNygRXWc4CLgVzr507gcQARiQceAOYCc4AHOhOMUkqdyDpvzAuGTu4hJwsRiQXOAp4CMMa0GmOqgUXAM9ZuzwBXWo8XAc8ajy8Al4ikAhcBy40xlcaYKmA5sHCocSmlVLAYFRMGwMHKRj9HMjBvahY5QDnwtIhsEJG/ikgkkGKMOWTtcxhIsR6nA4Xdji+yyvoqP4aI3Ckia0VkbXl5uRehK6WU/03NcBFiF1btrfR3KAPyJlk4gBnA48aY6UADXzY5AWA86wX6bM1AY8yTxphZxphZSUlJvjqtUkr5RbjTzrQMF1/sO7GTRRFQZIxZZT1/CU/yKLWal7B+l1nbi4HMbsdnWGV9lSul1Alv3pgEthbXUN/S7u9Q+jXkZGGMOQwUish4q+h8YDvwBtA5oulW4HXr8RvALdaoqHlAjdVctQxYICJxVsf2AqtMKaVOeHPHxNPhNqzdH9i1C4eXx38PeF5EnMBe4HY8CehFEbkDOABcZ+37DnAJUAA0WvtijKkUkZ8Da6z9HjTGBPZVU0opH5mZFYfDJqzaV8k545P9HU6fvEoWxpiNwKxeNp3fy74GuLuP8ywGFnsTi1JKBaMIp4OpGbF8sfeIv0Ppl97BrZRSfjZvTAJbimpoCOB+C00WSinlZ/PGJNDuNiz840f8+KVNHKlv8XdIx9BkoZRSfnbGuET++/KJTEyN4ZX1xfzpgwJ/h3QMTRZKKeVndptw2xk5PHHzLC6flsbL64oCrklKk4VSSgWQr83Loq6lndc2Du52sz3l9byyvoiW9o5hicvbobNKKaV8aMZoF5PSYvj75we4fvZo1h+soqS6ifqWdialxXJqpuuYY47Ut3DzX1dRUtPMw+/t5vvn53L1jHQcdt/VBzRZKKVUABERbp6XxX2vbGHOL9/nSEPPGWmnj3bx1VmZnJmbSEZcBO0dbu5ZuoGKhlZ+fuVkXlpbyI9f3syfP9zD3eeO46y8JGLDQ1i5q4xX1w99cgxNFkopFWAWnZrO0jWFJEWHcsW0NE5JjSHcaWf5tsP87bP93PfKFgBiw0NwOmyU17Xwm2umct3sTL42dzTLt5fy8Hu7+dE/NwEQFmKjuc1NYlTokGMSz71ywWfWrFlm7dq1/g5DKaVGlDGG/LJ6Psmv4MCRBuqa25maEcttZ+T02M/tNmworGb9gSoOVjZy3oRk5ucmEuKwrzPG9HYzdb+0ZqGUUkFERMhLiSYvJbrf/Ww2YWZWHDOzfLOWnI6GUkopNSBNFkoppQakyUIppdSANFkopZQakCYLpZRSA9JkoZRSakCaLJRSSg1Ik4VSSqkBBe0d3CJSB+zydxwDSAQq/B3EcQiGOIMhRtA4fSkYYoTgiLMzxgoAY8zCwZ4gmO/g3jWUW9ZHkoisDfQYITjiDIYYQeP0pWCIEYIjTl/EqM1QSimlBqTJQiml1ICCOVk86e8AjkMwxAjBEWcwxAgapy8FQ4wQHHF6HWPQdnArpZQaOcFcs1BKKTVCNFkopZQaUNAlCxFZKCK7RKRARO7zdzydRCRTRFaKyHYR2SYi37fK40VkuYjkW799sxKJd7HaRWSDiLxlPc8RkVXWNf2HiDgDIEaXiLwkIjtFZIeInBZo11JEfmD9W28VkSUiEhYI11JEFotImYhs7VbW67UTj0eseDeLyAw/x/lb6998s4i8KiKubtvut+LcJSIX+TPObtt+JCJGRBKt5365nn3FKCLfs67nNhH5TbfywV9LY0zQ/AB2YA8wBnACm4CJ/o7Lii0VmGE9jgZ2AxOB3wD3WeX3AQ8FQKw/BF4A3rKevwhcbz3+M/CdAIjxGeAb1mMn4AqkawmkA/uA8G7X8LZAuJbAWcAMYGu3sl6vHXAJ8C9AgHnAKj/HuQBwWI8f6hbnROvzHgrkWN8Ddn/FaZVnAsuAA0CiP69nH9fyXOB9INR6nuzNtRzR/8Q+uCCnAcu6Pb8fuN/fcfUR6+vAhXjuMk+1ylLx3Ezoz7gygBXAecBb1n/qim4f0B7X2E8xxlpfxHJUecBcSytZFALxeG5ufQu4KFCuJZB91BdHr9cOeAK4obf9/BHnUduuAp63Hvf4rFtf0qf5M07gJWAasL9bsvDb9ezl3/xF4IJe9hvStQy2ZqjOD2inIqssoIhINjAdWAWkGGMOWZsOAyl+CqvT/wA/BtzW8wSg2hjTbj0PhGuaA5QDT1vNZX8VkUgC6FoaY4qB3wEHgUNADbCOwLuWnfq6doH8mfo6nr/SIcDiFJFFQLExZtNRmwIpzjxgvtUs+qGIzLbKhxRjsCWLgCciUcDLwL3GmNru24wnjfttrLKIXAaUGWPW+SuG4+TAU6V+3BgzHWjA03TSJQCuZRywCE9iSwMigUHPt+MP/r52x0NEfgq0A8/7O5ajiUgE8J/A/+/vWAbgwFPznQf8B/CiiMhQTxZsyaIYTzthpwyrLCCISAieRPG8MeYVq7hURFKt7alAmb/iA84ArhCR/cBSPE1RfwRcItI5T1ggXNMioMgYs8p6/hKe5BFI1/ICYJ8xptwY0wa8guf6Btq17NTXtQu4z5SI3AZcBtxkJTYIrDjH4vkjYZP1WcoA1ovIKAIrziLgFeOxGk9rQiJDjDHYksUaINcaceIErgfe8HNMgGcUBPAUsMMY8/tum94AbrUe34qnL8MvjDH3G2MyjDHZeK7dB8aYm4CVwLXWbn6NEcAYcxgoFJHxVtH5wHYC6FriaX6aJyIR1r99Z4wBdS276evavQHcYo3imQfUdGuuGnEishBPM+kVxpjGbpveAK4XkVARyQFygdX+iNEYs8UYk2yMybY+S0V4BrccJrCu52t4OrkRkTw8A0UqGOq1HKkOIh924lyCZ6TRHuCn/o6nW1xn4qnabwY2Wj+X4OkTWAHk4xmZEO/vWK14z+HL0VBjrP8sBcA/sUZP+Dm+U4G11vV8DYgLtGsJ/AzYCWwF/o5ndInfryWwBE8/ShueL7I7+rp2eAY4PGp9nrYAs/wcZwGe9vTOz9Cfu+3/UyvOXcDF/ozzqO37+bKD2y/Xs49r6QSes/5/rgfO8+Za6nQfSimlBhRszVBKKaX8QJOFUkqpAWmyUEopNSBNFkoppQakyUIppdSANFkopZQakCYLpZRSA/p/pc3v3lSG4+YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "test_performance.net_worth.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When complete, strategy.run returns a `Pandas.data_frame` of the agent's performance, including the net worth and balance of the agent at each time step.\n",
    "\n",
    "## Live Trading\n",
    "\n",
    "Once you've built a profitable trading strategy, trained an agent to trade it properly, and ensured its \"generalize-ability\" to new data sets, all there is left to do is profit. Using a live exchange such as `CCXTExchange`, you can plug your strategy in and let it run!\n",
    "\n",
    "![Trading](img/trading.jpeg)\n",
    "\n",
    "While the gambler in you may enjoy starting a strategy and letting it run without bounds, the more risk averse of you can use a `trade_callback`, which will be called each time the strategy makes a trade. This callback function, similar to the episode callback, will pass in a data frame containing the agent's overall performance, and expects a `bool` in return. If `True`, the agent will continue trading, otherwise, the agent will stop and return its performance over the session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished running strategy.\n",
      "Total episodes: 0 (1 timesteps).\n",
      "Average reward: -0.5.\n"
     ]
    }
   ],
   "source": [
    "import ccxt\n",
    "from tensortrade.environments import TradingEnvironment\n",
    "from tensortrade.exchanges.live import CCXTExchange\n",
    "\n",
    "binance = ccxt.binance({\n",
    "    'apiKey': 'HfPX38sJ2aKewYDUJx6TaWrDhuT7rq426elO5Gbc55Dvg4klASEfm0aqcdl4Mpz6',\n",
    "    'secret': 'aGdMMaaol3GkyzDcrSEoc4aFlWG78qCxnGl3o22ub24u2scrZRcsIe2qn0kI82GQ',\n",
    "    'enableRateLimit': True,\n",
    "})\n",
    "\n",
    "exchange = CCXTExchange(exchange=binance,\n",
    "                        base_instrument='BNB',\n",
    "                        observation_type='ohlcv',\n",
    "                        timeframe='1h')\n",
    "\n",
    "btcusd_actions = DiscreteActions(n_actions=20, instrument='BNB/BTC')\n",
    "\n",
    "environment = TradingEnvironment(exchange=exchange,\n",
    "                                 feature_pipeline=feature_pipeline,\n",
    "                                 action_scheme=btcusd_actions,\n",
    "                                 reward_scheme=reward_scheme)\n",
    "\n",
    "strategy.environment = environment\n",
    "\n",
    "live_performance = strategy.run(steps=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "live_performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Passing `steps=0` instructs the strategy to run until otherwise stopped._\n",
    "\n",
    "That's all there is to it! As you can see, it is quite simple to build complex trading strategies using simple components and deep reinforcement learning. So what are you waiting for? Dive in, get your hands dirty, and see what's possible using TensorTrade.\n",
    "\n",
    "# Final Thoughts\n",
    "\n",
    "TensorTrade is a powerful framework capable of building highly modular, high performance trading systems. It is fairly simple and easy to experiment with new trading and investment strategies, while allowing you to leverage components from one strategy in another. But don't take my word for it, create a strategy of your own and start teaching your robots to take over the world!\n",
    "\n",
    "While this tutorial should be enough to get you started, there is still quite a lot more to learn if you want to create a profitable trading strategy. I encourage you to head over to the [Github](https://github.com/notadamking/tensortrade) and dive into the codebase, or take a look at our documentation at [tensortrade.org](https://tensortrade.org). There is also quite an active [Discord community](https://discord.gg/ZZ7BGWh) with nearly 1000 total members, so if you have questions, feedback, or feature requests, feel free to drop them there!\n",
    "\n",
    "![Commits](img/commits.png)\n",
    "\n",
    "I've gotten the project to a highly usable state. Though, my time is limited, and I believe there are many of you out there who could make valuable contributions to the open source codebase. So if you are a developer or data scientist with an interest in building state-of-the-art trading systems, I'd love to see you open a pull request, even if its just a simple test case!\n",
    "\n",
    "Others have asked how they can contribute to the project without writing code. There are currently three ways that you can do that. \n",
    "\n",
    "1. Write code or documentation for the TensorTrade framework. Many issues on the Github are funded through Gitcoin smart contracts, so you can actually get paid to contribute.\n",
    "\n",
    "2. Fund this project with either [Bitcoin](https://www.blockchain.com/btc/address/1Lc47bhYvdyKGk1qN8oBHdYQTkbFLL3PFw) or [Ethereum](https://www.blockchain.com/eth/address/0x9907A0cF64Ec9Fbf6Ed8FD4971090DE88222a9aC). These donations are used to fund our Gitcoin smart contracts, which allows anyone who contributes quality code and documentation to get paid for their contributions.\n",
    "\n",
    "3. Sponsor me on [Patreon](https://www.patreon.com/notadamking). Your support means a lot to me, and allows me to continue spending my time working on the framework and writing articles like this. All patrons have gotten early access to this article and codebase, and will continue to get early access to any articles I write in the future as a thank you.\n",
    "\n",
    "Thanks for reading! As always, all of the code for this tutorial can be found on my [GitHub](https://github.com/notadamking/tensortrade). Leave a comment below if you have any questions or feedback, I'd love to hear from you! I can also be reached on [Twitter](https://twitter.com/notadamking) at @notadamking.\n",
    "\n",
    "## References\n",
    "\n",
    "[1.] Introduction to Deep Reinforcement Learning Hui, Jonathan. \"RL- Introduction to Deep Reinforcement Learning.\" Medium, 7 Jan. 2019, https://medium.com/@jonathan_hui/rl-introduction-to-deep-reinforcement-learning-35c25e04c199.\n",
    "\n",
    "[2.] Policy Gradient Algorithms\n",
    "Weng, Lilian. \"Policy Gradient Algorithms.\" Lil'Log, 8 Apr. 2018, https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#reinforce.\n",
    "\n",
    "[3.] Clean Code: A Handbook of Agile Software Craftsmanship\n",
    "[Martin, Robert C. Clean Code: a Handbook of Agile Software Craftsmanship. Prentice Hall, 2010](https://amzn.to/2XANX1X).\n",
    "\n",
    "[4.] Advances in Financial Machine Learning\n",
    "[Prado Marcos López de. Advances in Financial Machine Learning. Wiley, 2018](https://amzn.to/2J6YCrW)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 2
}